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Abstract 

This  research  presents  a  methodology  to  evaluate  the  quality  of  a  system's 
architecture  using  principles  drawn  from  Value-Focused  Thinking  (VFT)  and  resulting  in 
a  Value-Driven  Enterprise  Architecture  Score  (VDEA-Score).  This  is  an  overall 
numerical  architecture  quality  score  useful  to  a  system's  management  team  to  identify  the 
advantages  and  disadvantages  of  a  system  design  and  associated  architecture 
documentation  or  to  track  its  quality  across  discrete  evaluation  epochs.  This  effort 
determined  which  aspects  of  the  architecture  are  most  valuable  to  the  stakeholder  in  the 
areas  of  (1)  the  system  effectiveness  values  (quality  of  the  instantiated  system  being 
represented  and  its  ability  to  perform  its  stated  mission)  and  (2)  the  architecture  quality 
values  (intrinsic  quality  of  the  products  themselves  in  terms  of  documentation  standards 
and  desired  attributes).  The  results  are  reported  across  three  theses.  In  this  thesis,  the 
architecture  documentation  quality  aspects  are  specifically  addressed  by  examining 
various  "ilities"  (e.g.,  usability,  modifiability,  accessibility,  etc.)  regarded  as  essential  to 
any  architecture.  The  evaluation  methodology  was  tested  against  architectures  from  two 
enterprises  including  the  sponsor's  enterprise  of  joint  force  protection.  An  overall 
architecture  documentation  quality  score  is  reported  for  both  enterprises  useful  for 
identifying  areas  for  potential  improvement. 
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VALUE-DRIVEN  ENTERPRISE  ARCHITECTURE  SCORE:  EVALUATION  APPLIED  TO 


JOINT  FORCE  PROTECTION  FUTURE  STATE  DESIGN 


I.  Introduction 


According  to  the  Defense  Science  Board  and  other  major  studies,  good  architectures  are  a 
key  to  good  interoperability  (DoD,  2007a:  1-1).  As  the  DoD  continues  its  transformation  to 
interoperable,  net-centric  systems  with  increasing  reliance  on  the  underlying  architecture 
descriptions  for  development,  the  authors  recognized  a  need  for  a  tool  to  assist  the  system’s 
management  team  in  evaluating  the  quality  of  their  system's  architecture.  The  authors  developed 
the  Value-Driven  Enterprise  Architecture  Score  (VDEA-Score)  to  identify  both  the  strengths  and 
areas  for  improvement  for  enhancing  the  usefulness  of  the  system's  architecture.  This  may  also 
serve  as  a  baseline  measure  to  compare  future  iterations  of  the  system's  architecture  through 
assessment  of  the  Architecture  Quality  and  System  Effectiveness  values. 

1.1.  General  Background 

The  U.S.  Clinger-Cohen  Act  of  1996  was  established,  in  part,  to  ensure  that  Department 
of  Defense  (DoD)  information  technology  (IT)  and  national  security  systems  were  interoperable. 
This  act  also  emphasized  a  great  need  for  joint  architectures  and  required  that  all  federal 
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government  chief  information  officers  "develop,  maintain,  and  facilitate  the  implementation  of  a 
sound  and  integrated  information  technology  architecture"  (U.S.  Congress,  1996).  As  the  DoD 
began  its  transformation  to  net-centric  warfare  (NCW),  the  importance  of  joint  architectures  to 
ensure  interoperability  was  highlighted.  The  requirement  for  architecture  development  was 
further  expounded  by  the  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS,  2007). 

The  DoD  Architecture  Framework  vl.5  (DoDAF)  is  the  means  to  interoperable 
architecture.  Consisting  of  29  products  (or  views)  representing  different  perspectives 
(operational  view  or  OV,  system  view  or  SV,  technical  view  or  TV,  and  all-view  or  AV),  it  aids 
in  the  system  design  and  serves  to  document  and  communicate  important  decisions  and 
problems.  Architectures  are  further  beneficial  to  “facilitate  decision  making  by  conveying  the 
necessary  information  to  the  decision  maker  for  the  decision  at  hand  as  well  as  enabling  the  reuse 
of  architecture  information  for  additional  needs”  (DoD,  2007a:3-l). 

Additionally,  the  Net-Centric  Enterprise  Solutions  for  Interoperability  (NESI)  provides 
voluntary  guidance  and  an  evaluation  checklist  for  NCW  programs.  This  cross-service  effort 
between  the  Air  Force,  Navy,  and  Defense  Information  Systems  Agency  (DISA)  guides  the 
design,  implementation,  maintenance,  evolution,  and  use  of  the  Information  Technology  (IT) 
portion  of  net-centric  solutions  for  defense  application  (NESI,  2008) 

1.2.  Specific  Background 

As  NCW  transforms  the  force  protection  domain,  interoperability  is  crucial  for  ensuring 
smooth  operations.  DoD  studies  have  shown  inadequacies  in  providing  comprehensive, 
integrated,  and  sustainable  joint  force  protection  capability  (Defense  Science  Board  Task  Force, 
2006).  Seeking  to  integrate  tactical  systems,  sensors,  and  security  personnel  to  protect  forces 
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while  promoting  interoperability  across  the  four  services,  the  DoD  created  the  Security 
Equipment  Integration  Working  Group  (SEIWG)  with  representatives  from  the  U.S.  Air  Force 
(USAF),  U.S.  Army  (USA),  U.S.  Navy  (USN),  and  the  U.S.  Marine  Corps  (USMC). 

The  SEIWG  domain  spans  the  DoD  and  shares  the  goal  of  interoperability  with  the 
Integrated  Unit,  Base  and  Installation  Protection  (IUBIP)  team  as  well  as  the  more  site-specific 
Joint  Force  Protection  Advanced  Security  System  Joint  Capability  Technology  Demonstration 
(JFPASS  JCTD).  The  JFPASS  JCTD  demonstrates  an  integrated  system-of-systems  to  protect 
military  installations,  incorporates  comprehensive  situational  awareness  for  force  protection 
providers,  reduces  manning  due  to  systems  integration  and  robotics,  and  reduces  logistics  cost. 
Functional  areas  for  installation  protection  addressed  include:  perimeter  security,  chemical- 
biological-radiological  defense,  access  control,  non-intrusive  inspection,  and  waterside  security. 

Within  the  SEIWG  mission  to  “coordinate  and  influence  system  architecture,  technical 
design,  and  systems  integration”  (Havlicek,  2008),  the  working  group  is  working  to  improve 
interoperability  by  developing  the  “to-be”  architecture  for  joint  net-centric  force  protection 
within  the  DoDAF  and  NESI  guidelines.  These  architecture  views  are  intended  to  cover  the 
Detect,  Assess,  Warn,  Defend,  and  Recover  (DAWDR)  activities;  be  suitable  for  inclusion  in  a 
Joint  Protection  Capability  Development  Document  (CDD);  and  provide  guidance  to  all  services 
ensuring  interoperability  of  force  protection  systems. 

1.3.  Research  Problem 

The  Air  Force’s  642d  Electronic  Systems  Squadron  (ELSS),  as  the  current  chair  of  the 
SEIWG,  solicited  AFIT’s  help  in  researching  joint  force  protection  values  with  measures  and  a 
framework  to  evaluate  the  quality  of  their  proposed  “to-be”  architecture.  Satisfying  this  need 
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will  provide  better  insight  into  the  important  factors  impacting  the  overall  joint  force  protection 


process. 

1.4.  Research  Objective 

To  complete  this  research  problem,  the  objective  was  two-fold.  The  first  aspect  was 
developing  a  reliable  and  repeatable  model  to  evaluate  the  quality  of  any  DoDAF  architecture. 
The  second  was  to  apply  this  model  using  common  joint  force  protection  values  to  evaluate  a 
“To-Be”  architecture  for  net-centric  force  protection  resulting  in  an  overall  value  score.  The 
following  investigative  questions  were  addressed  during  this  research. 

1 .  What  are  the  “best”  methods  to  evaluate  and  measure  the  overall  quality  of  an 
architecture? 

2.  What  are  the  major  categories  and  sub-categories  that  should  be  considered  when 
evaluating  an  architecture? 

3.  What  are  the  major  categories  and  sub-categories  that  should  be  considered  when 
evaluating  force  protection  processes? 

4.  How  do  these  categories  and  sub-categories  rank  in  terms  of  importance? 

5.  How  well  does  current  JFPASS  architecture  meet  the  weighted  values  of  the  force 
protection  community? 

1.5.  Methodology 

Developed  in  1992  by  Keeney  (1994)  and  refined  in  1996  by  Kirkwood  (1997),  Value- 

Focused  Thinking  (VFT)  is  a  decision  analysis  tool  that  organizations  have  successfully  used  to 

make  decisions  and  is  a  natural  fit  to  the  force  protection  value  problem.  The  antithesis  to  usual 

alternative-based  approaches  to  decision  making,  VFT  provides  an  opportunity  for  proactive 
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decision  making  that  focuses  on  objectives,  as  opposed  to  reactive  decision  making  that  focuses 
on  means.  VFT  has  been  employed  in  a  wide  range  of  areas  such  as  climate  change  research, 
nuclear  waste  transportation,  and  public  health  in  the  mining  industry  (Kirkwood,  1997). 

More  specifically,  the  AFIT-developed  10-Step  VFT  Process  as  reported  by  several 
authors  such  as  Shoviak  (2001),  Jurk  (2002),  and  Braziel  (2004)  was  used  to  guide  the  VDEA- 
Score  development.  This  effort  determined  which  aspects  of  the  architecture  are  most  valuable 
to  the  stakeholder  in  the  areas  of  (1)  the  system  effectiveness  values  (quality  of  the  instantiated 
system  being  represented  and  its  ability  to  perform  its  stated  mission)  and  (2)  the  architecture 
quality  values  (intrinsic  quality  of  the  products  in  terms  of  documentation  standards  and  desired 
attributes).  These  values  formed  the  model  used  to  evaluate  the  “To-Be”  force  protection 
architecture. 

1.6.  Research  Scope 

The  overall  research  effort  was  divided  between  this  thesis  (specifically  focused  on  the 
architecture  product  quality  values)  and  the  work  of  Osgood  (2009)  and  Mills  (2009)  on  the 
force  protection  focused  system  values.  This  thesis  examined  various  “ilities”  (e.g.,  usability, 
modifiability,  accessibility,  etc.)  regarded  as  vital  to  any  architecture.  This  effort  began  by 
taking  a  generic  perspective  to  thus  enable  the  reuse  of  the  value  categories,  definitions,  and 
measures  to  other  projects.  It  was  then  tailored  to  joint  force  protection  by  applying  weighting 
factors  according  to  how  the  sponsor  valued  each  category.  An  overall  architecture  quality  score 
was  then  derived  as  a  reference  point.  But  more  specifically,  the  score  was  used  to  identify  areas 
of  improvement.  Finally,  as  a  reference  point  to  validate  the  value  categories  (definitions  and 
measures),  a  subsequent  evaluation  of  another  program’s  architecture  views  was  conducted. 
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1.7.  Chapter  Overview 


Chapter  2  presents  a  literature  review  that  provides  insight  into  force  protection, 
architectures,  and  other  architecture  evaluation  methods.  The  decision  analysis  methodology  and 
VFT  process,  as  well  as  their  relevance  to  this  research,  will  also  be  discussed  in  Chapter  2. 
Chapter  3  discusses  the  methodology  used  to  determine  the  architecture  value  hierarchy, 
definition  of  the  values  used,  how  these  values  are  measured  using  VFT,  and  how  these  values 
were  weighted  to  enable  evaluation.  The  analysis  of  the  value  model  is  provided  in  Chapter  4  by 
evaluating  a  “To-Be”  architecture  for  force  protection  to  judge  its  quality  and  effectiveness, 
identify  any  deficiencies  in  the  value  model,  and  create  a  composite  value-focused  joint  force 
protection  score.  Chapter  5  summarizes  the  research  results  and  proposes  conclusions  and 
recommendations  for  future  research. 
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II.  Background 


This  chapter  provides  background  information  on  joint  force  protection  and  quality 
system  architecting.  The  chapter  then  continues  with  an  examination  of  decision  analysis 
methodology,  culminating  in  the  value-focused  thinking  (VFT)  approach  for  determining  a 
degree  of  quality  leading  to  the  concept  of  a  VDEA-Score  for  architecture  quality.  It  addresses 
published  information  on  system  architecting  and  more  specifically  quality  attributes  referred  to 
as  the  “ilities.”  Finally,  a  review  of  information  available  regarding  approaches  to  quantifying 
these  attributes  is  included. 

2.1.  Joint  Force  Protection 

Force  protection  is  specifically  identified  in  the  National  Military  Strategy  (NMS).  The 
NMS  specifies,  “The  Armed  Forces  must  have  the  ability  to  operate  across  the  air,  land,  sea, 
space  and  cyberspace  domains  of  the  battlespace.  Armed  Forces  must  employ  military 
capabilities  to  ensure  access  to  these  domains  to  protect  the  Nation,  forces  in  the  field  and  US 
global  interests ”  (emphasis  added,  CJCS,  2004). 

Although  very  general  and  focused  on  implementation  by  2015,  the  Protection  Joint 
Functional  Concept  (PJFC)  provides  the  next  level  of  guidance.  By  way  of  definition,  the  PJCF 
states  that: 

protection  is  a  process,  set  of  activities,  or  utilization  of  capabilities  by  which  the  joint 
force  protects  personnel  (combatant/non-combatant),  physical  assets,  and  information  of 
the  United  States,  allies  and  friends,  required  to  ensure  fighting  potential  can  be  applied 
at  the  decisive  time  and  place  against  the  full  spectrum  of  threats.  (J8,  2004:7) 

The  PJCF  recognizes  that  “current  protection  efforts  are  characterized  by  channelized  and 

sometimes  conflicting  efforts... [which]  ...could  create  wasteful  and  potentially  harmful  technical 
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and  operational  gaps”  (J8,  2004:  9).  To  combat  these  technical  and  operational  gaps,  the  PJFC 
specifies  that  “execution  of  protection  operations  in  2015  must  be  integrated  with  the 
overarching  Joint  Force  operations  construct”  (J8,  2004:8)  as  depicted  in  Figure  1. 


1  Conduct  Force 

Operations  | 

[^  [Monitor)  |  Understand)  |  Decide 

_ |  Execute)  J 

7 7 7  7\ 


"V" 


Mission  Capability  Elements 


•Air  and  Missile  Defense 
•IED  Defense 
•Combating  WMD 
•Force  Health  Protection 


•Maritime  Defense 
•Defensive  Counter-Space 
•Non  Combatant  Evacuation 
•Others... 


Figure  1.  The  Protection  Construct  (J8,  2004:  8) 


Therefore,  for  integrated  joint  forces,  interoperability  is  the  key  doctrinal  idea  to  enable 
operations  in  a  joint  environment.  The  Universal  Joint  Task  List  (UJTL)  is  the  next  level  of 
guidance  giving  each  service  its  specific  missions  and  areas  of  responsibility.  The  UJTL 
defines  interoperability  as  “the  ability  of  systems,  units,  or  forces  to  provide  service  to  and 
accept  services  from  other  systems,  units,  or  forces  and  to  use  the  services  so  exchanged  to 
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enable  them  to  operate  effectively  together”  (CJCS,  2002:  A-5).  While  the  UJTL  addresses 
interoperability  and  specifies  which  portions  of  the  mission  each  service  will  do,  how  to 
actually  implement  this  is  not  specified.  Thus,  each  service  is  allowed  to  implement  it 
differently.  As  more  instances  of  joint  basing  occur,  especially  in  deployed  locations  in  which 
Air  Force  security  personnel  are  augmenting  other  service’s  forces,  interoperability  is  crucial  for 
ensuring  smooth  operations. 

With  the  SEIWG’s  mission  to  “coordinate  and  influence  system  architecture,  technical 
design,  and  systems  integration”  (Havlicek,  2008:  2),  it  is  working  to  improve  interoperability  by 
developing  the  “to-be”  architecture  for  joint  net-centric  force  protection.  As  the  current  chair  of 
the  SEIWG,  the  642d  ELSS  solicited  AFIT’s  help  in  researching  joint  force  protection  values 
with  measures  and  a  framework  to  evaluate  the  quality  of  their  proposed  “to-be”  architecture 
(Havlicek,  2008:2). 

2.2.  Decision  Analysis 

The  SEIWG  faces  hard  decisions  accomplishing  its  mission  across  DoD  force  protection 
stakeholders.  Clemen  (2001 :2-3)  identifies  four  sources  of  difficulty  in  making  a  decision.  First 
is  complexity.  Many  issues,  possible  courses  of  action,  possible  outcomes,  etc.,  may  be  almost 
impossible  to  keep  straight  at  one  time  and  require  organization  and  analysis.  Secondly,  the 
uncertainty  inherent  in  the  situation  may  prove  difficult.  Thirdly,  multiple  objectives  (especially 
if  an  advance  toward  one  causes  problems  with  another)  may  drive  tradeoffs  in  benefit  and  cost 
between  objectives.  Finally,  differing  perspectives  or  inputs  can  drive  differing  conclusions  or 
choices.  This  is  especially  poignant  in  the  joint  force  protection  arena  when  consulting 
stakeholders  across  the  DoD  whose  approaches  may  be  significantly  different  from  each  other. 
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The  concepts  of  decision  analysis  exist  to  provide  “structure  and  guidance  for  thinking 
systematically  about  hard  decisions”  (Clemen,  2001:2).  Two  main  approaches  in  thinking  are 
found  in  literature — alternative-focused  thinking  (AFT)  and  value-focused  thinking  (VFT). 

2.3.  Alternative  vs.  Value-Focused  Thinking 

The  differences  between  Alternative-Focused  Thinking  and  Value-Focused  Thinking  are 
straightforward.  From  an  AFT  perspective,  the  decision  maker  identifies  the  problem  and  then 
compares  the  alternatives  available  for  solving  it.  VFT,  on  the  other  hand,  focuses  more  on 
what  is  important  or  valued  by  the  decision  maker,  who  then  explores  ways  to  achieve  the  best 
value.  Keeney  (1993:3)  pointedly  describes  the  difference  between  the  two  as:  “[values]  are 
fundamentally  important  in  any  decision  situation.  Alternatives  are  relevant  only  because  they 
are  the  means  to  achieve  your  values.” 

Instead  of  primarily  looking  at  available  alternatives,  the  goal  of  VFT  is  to  create  a 
mutually-exclusive  and  collectively-exhaustive  set  of  values  which  contain  all  the  important 
points  to  the  decision  maker  (Kirkwood,  2007:17).  This,  in  turn,  leaves  the  door  open  to  potential 
undiscovered  alternatives  which  may  prove  more  beneficial  in  reaching  the  desired  objectives. 
Shoviak  (2001:46)  provided  a  good  table  summary  of  the  advantages  of  VFT  as  shown  in  Table 
1  followed  by  Jurk’s  (2002:27)  synopsis  of  key  VFT  terminology  in  Table  2. 


10 


Table  1.  Advantages  of  VFT  (Shoviak,  2001:46) 


Advantage 

Description 

Uncovering  hidden 
objectives 

Value-focused  thinking  includes  a  number  of  techniques 
that  can  be  used  to  stimulate  creativity  in  identifying 
possible  objectives  not  yet  realized. 

Creating  alternatives 

Focusing  on  the  values  that  should  be  guiding  the  decision 
makes  the  search  for  new  alternatives  a  creative  and 
productive  exercise  (Keeney,  1994:  39).  Creating  new 
alternatives  may  be  more  important  than  evaluating 
available  alternatives. 

Identifying  decision 
opportunities 

Decision  situations  should  be  viewed  as  opportunities  to 
take  advantage  of  and  not  as  problems  to  solve. 

Systematically  evaluating  whether  and  how  to  better  achieve 
your  values  may  create  decision  opportunities. 

Guiding  strategic  thinking 

Value-focused  thinking  compels  the  decision-maker  to 
formulate  strategic  objectives. 

Inter-connecting  decisions 

“Strategic  objectives  provide  common  guidance  for  all 
decisions  in  an  organization  and  form  the  basis  for  more 
detailed  fundamental  objectives  appropriate  for  specific 
decisions”  (Keeney,  1994:  34). 

Guiding  information 
collection 

When  what  is  important  to  the  decision  situation  is  known, 
then  one  can  be  sure  to  collect  information  about  the 
important  objectives  and  not  waste  valuable  resources 
collecting  information  about  objectives  that  are  not 
important. 

Facilitating  improvement 
in  multiple-stakeholder 
decisions 

Many  decisions  involve  multiple  stakeholders  who  have 
their  own  interests  in  the  decision.  Value-focused  thinking 
helps  to  facilitate  communications  among  the  stakeholders 
regarding  the  important  objectives  for  decision.  “In 
situations  with  controversy,  a  common  understanding  about 
what  are  important  [objectives]  may  provide  a  better  basis 
for  compromise  and/or  consensus  with  regard  to  selecting 
alternatives”  (Kirkwood,  1997:  23). 

Improving  communication 

Value-focused  thinking  uses  a  common  vocabulary 
regarding  the  achievement  of  objectives  in  a  particular 
decision  context.  This  basis  should  help  facilitate 
communication  and  understanding. 

Evaluating  alternatives 

Value-focused  thinking  provides  a  framework  for 
quantifying  values,  which  allows  one  to  construct  a 
quantitative  value  model  to  evaluate  various  alternatives  and 
rank  them  by  total  value.  Sensitivity  analysis  of  an 
alternative’s  desirability  to  a  specific  value  may  be 
conducted  to  provide  the  decision-maker  powerful  insight. 
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Table  2.  Key  VFT  Terminology  (Jurk,  2002:27) 


Fundamental  Objective 

.  .an  essential  reason  for  interest  in  the 
decision  situation”  (Keeney,  1992:34).  Also 
known  as  the  “ends  objective,”  it  is  the  top 
block  in  the  value  hierarchy. 

Value 

What  is  important  to  the  decision  maker 
(Clemen,  1996:19).  The  values  are  the 
decomposition  of  the  fundamental  objective. 
They  are  the  building  blocks  of  the  value 
hierarchy. 

Value  Hierarchy 

A  pictorial  representation  of  a  value  structure 
(consisting  of  the  fundamental  objective,  the 
values,  and  the  measures)  (Kirkwood, 

1997:12). 

Local  Weight 

The  amount  of  weight  a  set  of  lower-tier  values 
or  measures  contributes  to  the  value  directly 
above  it  in  the  hierarchy  (Shoviak,  2001:57) 

Global  Weight 

The  amount  of  weight  each  lower-tier  value  or 
measure  contributes  to  the  weight  of  the 
hierarchy’s  fundamental  objective  (Shoviak, 
2001:57). 

Measure 

Analogous  to  the  term  “metric,”  it  notes  the 
“degree  of  attainment”  of  a  value  (Kirkwood, 
1997:12). 

Score 

A  “specific  numerical  rating  for  a  particular 
alternative  with  respect  to  a  specified  measure” 
(Kirkwood,  1997:12). 

Single  Dimensional  Value  Function  (SDVF) 

A  specific,  monotonically  increasing  or 
decreasing  function  for  each  measure  used  to 
convert  an  alternative’s  “score”  on  the  x-axis 
to  a  “value”  on  the  y-axis. 

Alternative 

“. .  .the  means  to  achieve  the  . .  .values” 

(Keeney,  1992:3) 
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In  determining  the  values,  Burk  (1997),  Parnell  (2007),  Knighton  (2007),  and  Dawley  et 
al.,  (2008)  describe  three  standards  of  sources:  platinum,  gold,  and  silver.  In  order  of  preference, 
platinum  comes  first  by  using  interviews  with  senior  stakeholders  and  the  actual  decision  maker 
to  determine  the  values.  Gold  is  next  using  published  policy  or  strategic  planning  documents 
approved  by  the  decision  maker.  Least  desirable  is  silver,  which  relies  on  interviews  with 
subject  matter  experts  (SMEs)  and  stakeholder  representatives.  These  standards  may  also  be 
combined.  “For  example,  we  could  combine  a  review  of  several  gold-standard  documents  with 
findings  from  interviews  with  decision  makers  and  stakeholders”  (Parnell,  2007). 

For  this  effort,  the  SEIWG  is  not  comparing  competing  architectures  but  rather 
comparing  against  today’s  performance.  The  SEIWG  is  developing  a  future  “to-be”  architecture 
that  reflects  the  important  aspects  of  force  protection.  Therefore,  the  VFT  approach  is  the  most 
appropriate  for  this  effort. 

2.4.  Ten-Step  VFT  Process 

A  number  of  research  efforts  have  benefited  from  this  VFT  approach  by  applying  the 
following  10-step  process  developed  at  AFIT  and  reported  by  several  authors  such  as  Shoviak 
(2001),  Jurk  (2002),  and  Braziel  (2004).  This  process,  shown  graphically  in  Figure  2,  was 
derived  from  the  methodology  described  by  Keeney  (1992)  and  Kirkwood  (1997).  The  authors 
used  this  10-Step  VFT  process  to  guide  the  VDEA-Score  methodology  development. 

2.4.1.  Step  1:  Problem  Identification 

This  VFT  process  begins  with  the  most-important  aspect  of  understanding  the  context  of 
the  decision  by  clearly  identifying  the  problem.  Identifying  the  wrong  problem  leads  to  wasted 
effort  and  what  Clemen  refers  to  as  an  “error  of  the  third  kind”  (2001 :5).  Braziel  (2004:27) 
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suggests  that  the  decision  maker  should  ask  two  questions:  “What  is  important  to  me  in  terms  of 
this  decision?  What  is  it  that  I  value  in  a  solution?”  Answering  these  may  help  properly  identify 
the  problem  and  lead  to  the  beginning  construction  of  a  value  hierarchy. 


Figure  2.  VFT  Ten-Step  Process  (Shoviak,  2001:63) 


14 


2.4.2.  Step  2:  Create  Value  Hierarchy 


With  the  problem  identified,  the  value  hierarchy  can  be  constructed  as  a  graphical 
representation  of  the  important  values.  This  allows  the  decision  maker  or  stakeholders  to  better 
visualize  the  values  and  help  identify  any  missing  values  or  “holes”  which  need  to  be  filled 
(Keeney,  1992:69).  When  creating  the  value  hierarchy,  five  desirable  properties  exist  that  should 
be  kept  in  mind:  completeness,  nonredundancy,  decomposability,  operability,  and  small  size 
(Kirkwood:  1997:16-18).  Table  3  describes  these  properties. 


Table  3.  Value  Hierarchy  Desired  Properties  (Kirkwood:  1997:16-18) 


Desired  Property 

Description 

Completeness 
(or  “collectively- 
exhaustive”) 

The  values,  when  taken  together  as  a  group  at  each  tier,  appropriately 
addresses  all  the  values  for  evaluating  the  overall  objective  of  the 
decision. 

Nonredundancy  (or 
“mutually  exclusive”) 

No  values  in  the  same  tier  overlap. 

Decomposability  (or 
“independence”) 

The  score  from  one  value’s  measure  does  not  depend  on  the  score  of 
another. 

Operability 

The  hierarchy  is  understandable  for  those  who  may  use  it 

Small  Size 

The  hierarchy  easier  to  communicate  to  stakeholders  and  uses  few 
resources. 

2.4.2.I.  Generating  Values 

To  generate  the  objectives  or  values  which  are  important  to  the  decision  problem, 
Shoviak  (2001:48)  provides  the  following  list  of  questions  based  on  techniques  Keeney 
developed  (1994:35). 

1 .  Develop  a  wish  list.  What  do  you  want?  What  do  you  value?  What  should 
you  want? 
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2.  Identify  alternatives.  What  is  a  perfect  alternative,  a  terrible  alternative, 
some  reasonable  alternative? 

3.  Consider  problems  and  shortcomings.  What  needs  fixing? 

4.  Predict  consequences.  What  has  occurred  that  was  good  or  bad?  What 
might  occur  that  you  care  about? 

5.  Identify  goals,  constraints,  and  guidelines.  What  are  your  aspirations? 

What  limitations  are  placed  on  you? 

6.  Consider  different  perspectives.  What  would  your  competitor  or 
constituency  be  concerned  about?  At  sometime  in  the  future,  what  would 
concern  you? 

7.  Determine  strategic  objectives.  What  are  your  ultimate  objectives?  What 
are  your  values  that  are  absolutely  fundamental? 

8.  Determine  generic  objectives.  What  objectives  do  you  have  for  your 
customers,  your  employees,  your  shareholders,  and  yourself?  What 
environmental,  social,  economic,  or  health  and  safety  objectives  are 
important? 

2.4.2.2.  Structuring  Values 

The  value  hierarchy  or  tree  is  constructed  to  show  how  the  values  relate  to  each  other.  At 
the  top  of  the  tree  is  the  most-general  but  highest-level  objective.  This  tree  can  be  further 
divided  down  to  lower  levels  or  tiers  where  the  lower-level  values  more  specifically  describe  the 
higher-level  objectives  or  values.  This  iterative  decomposition  of  general  (higher-level)  values 
into  more  specific  (lower-level)  values  continues  until  “the  values  are  subdivided  to  a  level  at 
which  measurement  and  evaluation  is  possible”  (Braziel  2004:32).  Jurk  (2002:36)  provided  the 
example  in  Figure  3  to  help  illustrate  this  concept  using  “Buy  the  Best  Truck”  as  the  highest 
level  objective  with  performance,  practicality,  and  safety  as  the  first-tier  values. 
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2.4.3.  Step  3:  Develop  Evaluation  Measures 


After  the  value  hierarchy  is  built  such  that  the  lowest  tier  has  the  most  specific  values, 
one  or  more  measures  are  developed  for  each  bottom-tier  value.  These  measures  are  the  means 
of  determining  the  extent  to  which  value  is  earned.  Referring  again  to  the  Jurk  (2001 :3  8) 
example  in  Figure  3,  an  example  measure  for  the  "Power"  value  could  be  "Horsepower." 

Measures  can  be  classified  as  either  natural  or  constructed  and  direct  or  proxy.  As  the 
name  implies,  a  natural  measure  is  widely  used  and  understood,  such  as  "horsepower"  from  the 
example.  A  constructed  measure,  on  the  other  hand,  is  created  to  address  a  particular  issue  when 
a  natural  measure  is  unavailable  or  inappropriate.  In  terms  of  direct  or  proxy,  a  "direct  scale 
directly  measures  the  degree  of  attainment  of  an  objective,  while  a  proxy  scale  reflects  the  degree 
of  attainment  of  its  associated  objective,  but  does  not  directly  measure  this"  (Kirkwood  1997:24). 
Therefore,  a  natural  and  direct  measure  is  the  goal  while  trying  to  minimize  the  use  of 
constructed  and  proxy  measures. 

Keeney  (1992: 112-116)  further  points  out  three  properties  desirable  for  an  evaluation 
measure:  measurability,  operationality,  and  understandability.  Measurability  means  the  specific 
measure  "must  clearly  and  appropriately  quantify  what  the  decision-maker  is  interested  in  and 
nothing  more"  (Braziel  2004:38).  The  operationality  property  "express(es)  relative  preferences 
for  different  levels  of  achievement  of  an  objective  as  indicated  by  attribute  levels"  (Keeney, 
1992:1 14).  Finally,  understandability  strives  to  eliminate  ambiguities  so  "no  loss  of  information 
[occurs]  when  one  person  assigns  [a  measure]  level  to  describe  a  consequence  and  another 
person  interprets  that  level"  (Keeney,  1992:116). 
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Figure  3.  “Buy  the  Best  Truck”  Hierarchy  (Jurk,  2002:36) 
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2.4.4.  Step  4:  Create  Value  Functions 


With  the  evaluation  measures  determined,  a  value  function  for  each  measure  must  be 
created.  Because  each  measure  may  have  a  completely  different  unit  or  scale,  simply  summing 
all  the  evaluated  measures  does  not  result  in  a  useful  overall  score.  Hence,  the  Single 
Dimension  Value  Function  (SDVF)  converts  each  measure  into  a  common  "value  unit"  between 
zero  and  one  where  "the  least  preferred  score  being  considered  for  a  particular  evaluation 
measure  will  have  a  single  dimensional  value  of  zero,  and  the  most  preferred  score  will  have  a 
single  dimensional  value  of  one"  (Kirkwood,  1997:61). 

The  SDVF  is  best  viewed  as  a  graph  created  by  an  x  and  y-axis.  The  range  of  points 
encompassing  the  specific  evaluation  measure  forms  the  x-axis.  The  value  score  (0  to  1)  is 
placed  on  the  y-axis.  Therefore,  the  decision  maker  determines  the  corresponding  value  of  each 
measure  based  on  the  function  created. 

Three  primary  types  of  SDVFs  are  discrete,  piecewise  linear,  and  exponential.  The 
piecewise  linear  function  "is  made  up  of  segments  of  straight  lines  that  are  joined  together" 
whereas  the  exponential  "uses  a  specific  mathematical  form"  (Kirkwood  1997:61).  These 
SDVFs  may  also  be  either  monotonically  increasing  (increased  y-axis  value  for  an  increased  x- 
axis  score)  or  decreasing  (decreased  y-axis  value  for  an  increased  x-axis  score).  Examples  of 
these  are  shown  in  Figures  4-6.  Figure  4  shows  a  discrete  SDVF  where  each  successive 
evaluation  category  earns  more  value.  In  Figure  5,  SDVF  1  shows  a  decreasing  rate  of  value 
earned  for  increased  evaluation  score.  SDVF  2  shows  a  constant  increase  in  value  with  increased 
evaluation  score.  SDVF  3  shows  an  increasing  value  rate  for  increased  evaluation  score.  In 
Figure  6,  SDVF  1  shows  a  slower  decreasing  rate  of  value  lost  with  increased  evaluation  score. 
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SDYF  2  shows  a  constant  decrease  in  value  with  increased  evaluation  score.  SDYF  3  shows  an 


increasing  rate  of  value  lost  with  increased  evaluation  score. 


Value 


Figure  4.  Discrete  or  Categorical  Functions  (Jurk,  2002:43) 
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Example  Monotonically  Increasing  Functions  (Kirkwood,  1997) 


Figure  6.  Example  Monotonically  Decreasing  Functions  (Kirkwood,  1997) 
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2.4.5.  Step  5:  Weight  Value  Hierarchy 


Because  all  value  categories  are  not  equal  in  the  eye  of  the  decision  maker,  each  one 
should  be  considered  against  each  other  in  terms  of  its  importance  after  creating  the  value 
functions.  The  decision  maker  assigns  a  weight  to  each  value  as  a  portion  of  the  total  weight  of 
the  hierarchy  which  when  summed  equals  one.  Continuing  with  Jurk's  (2002:45)  truck  example 
as  shown  in  Figure  7,  the  top  of  the  hierarchy  ("Buy  the  Best  Truck")  has  a  total  weight  of  one. 
For  the  three  values  on  the  second  tier,  the  weight  of  these  values  is  determined  by  considering 
their  importance  against  one  another  within  the  same  branch  and  tier  (called  the  local  weight) 
which  likewise  sums  to  one.  This  is  repeated  for  each  branch  and  tier  until  each  value  has  a  local 
weight. 

Now  that  each  value  has  a  local  weight,  a  global  weight  is  determined  which  shows  each 
value's  relative  importance  in  the  overall  hierarchy.  Katzer  (2002:4)  explains  this  is 
accomplished  by  "multiplying  the  local  weights  for  each  successive  tier  above  it."  Figure  8 
illustrates  the  overall  global  weights  applied  to  the  "Buy  the  Best  Truck"  example. 
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Figure  7.  “Buy  the  Best  Truck”  Example  with  Local  Weights  (Jurk,  2002:45) 
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Figure  8.  “Buy  the  Best  Truck”  Example  with  Global  Weights  (Jurk,  2002:49) 
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2.4.6.  Step  6:  Alternative  Generation 


With  the  value  hierarchy  appropriately  weighted,  potential  alternatives  may  be  generated 
which  meet  the  decision  need.  Regarding  these  alternatives,  Keeney  (1992: 198)  points  out  that 
"alternatives  should  be  created  that  best  achieve  the  values  specified  for  the  decision 
situation... [and  these]  alternatives  themselves  can  trigger  thought  processes  that  generate  new 
alternatives."  Braziel  (2004:39)  points  out  that  the  value  functions  of  the  hierarchy  act  as  a 
"screening  criterion."  If  too  many  alternatives  are  presented,  those  scoring  zero  against  the 
values  may  easily  be  removed.  On  the  other  hand,  if  not  enough  alternatives  present  themselves, 
then  the  "hierarchy  can  identify  value  gaps. ..[which  are]  instrumental  in  modifying  the  hierarchy 
in  order  for  alternatives  to  score  better  in  critical  areas"  (Braziel  2004:39). 

2.4.7.  Step  7:  Alternative  Scoring 

After  the  alternatives  to  be  evaluated  are  presented,  each  one  is  evaluated  according  to 
the  measures  for  each  value.  The  result  from  each  measure  is  then  applied  to  the  SDVF  for  a 
value  score.  Depending  on  the  number  of  measures  and  the  number  of  alternatives,  this  may  be  a 
lengthy  step. 

2.4.8.  Step  8:  Deterministic  Analysis 

With  the  score  for  each  value  determined,  the  associated  weights  are  next  applied 
resulting  in  the  weighted  sum  score  providing  the  means  to  rank  order  the  alternatives.  The 
additive  value  function  is  the  frequently  used  decision  analysis  mathematical  equation  for  this 
rank  ordering  (Braziel  2004:40).  Assuming  the  prerequisites  were  in  place  from  the  previous 
steps  (SDVF  with  values  between  zero  and  one  and  weighted  such  that  the  combined  weights  for 
an  alternative  sums  to  one),  the  general  additive  value  function  is  (Kirkwood,  1997:230): 
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v (*)  = 
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n 

and  ^ '  Aj  =  1 

i=i 

Where, 

v(x)  =  overall  score 
Vi(Xi)  =  score  value  of  the  ithmeasure, 

Aj  =  weight  of  the  ithmeasure, 

n=  £  1 

Vie{measures } 

Shoviak  (2001:60)  further  points  out  that  this  function  does  not  take  into  account  any  interaction 
with  any  other  alternatives.  This  preferential  independence  condition  therefore  implies  "that  the 
decision-maker's  preferences  associated  with  any  one  objective  are  independent  of  the  evaluation 
measure  scores  associated  with  all  other  objectives"  (Shoviak  2001:60). 

2.4.9.  Step  9:  Sensitivity  Analysis 

As  additional  insight  for  the  decision  maker,  analyzing  the  sensitivity  of  the  previous 
rank  ordering  can  be  accomplished  by  changing  the  assigned  weightings.  Because  there  is  little- 
to-no  change  in  the  SDVFs,  the  weight  of  each  value  is  varied  systematically  while  maintaining 
the  other  value  weightings  proportionally  the  same.  The  resulting  effect  on  the  overall  score  and 
rankings  can  be  tracked  to  provide  the  decision  maker  insight  to  the  impact  the  weightings  may 
have  on  the  choice  of  alternative. 
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2.4.10.  Step  10:  Conclusions  &  Recommendations 


Finally,  all  these  results  are  presented  to  the  decision  maker.  This  objective  ranking 
serves  as  a  supporting  tool  to  solving  the  decision  problem.  The  decision  maker  may  make  a 
better  informed  decision  with  the  aid  of  these  results. 

2.5.  Architectures 

2.5.1.  Definitions  of  Architecting,  Benefits,  Growth,  and  Guidance 

Over  the  past  decade,  the  field  of  systems  engineering  with  its  holistic  approach  to 
dealing  with  increasingly  complex  systems  has  grown  tremendously.  An  important  tool  in  the 
system  engineer’s  toolbox  is  the  system  architecture.  While  there  are  many  different  definitions 
for  system  architecture,  the  DoD  Architecture  Framework  (DoDAF)  definition  is:  “The  structure 
of  components,  their  relationships,  and  the  principles  and  guidelines  governing  their  design  and 
evolution  over  time"  (DoD,  2007a:ES-l).  Hence  the  fundamental  purpose  behind  the 
architecture  is  to  deconstruct  the  complex  system  into  an  easier-to-understand  representation  of 
the  system. 

Architectures  are  used  for  a  variety  of  purposes  which  include  supporting  strategic 
planning,  identifying  capability  needs,  relating  needs  to  systems  development  and  integration, 
and  facilitating  interoperability  and  supportability  (DoD,  2007a:3-l).  They  are  further  valuable 
in  aiding  the  decision  maker  by  providing  pertinent  information  associated  with  each  of  those 
purposes.  They  can  also  be  used  at  different  portfolio  levels  as  described  in  DoDAF  vl.5  (DoD, 
2007a:  3-1): 

•  Enterprise  -  Architectures,  particularly  federated  architectures,  are  used  at  the  enterprise 
level  to  make  better  decisions  that  improve  (1)  human  resource  utilization,  (2) 
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deployment  of  assets,  (3)  warfighter  investments,  and  (4)  identification  of  the  enterprise 
boundary  (interfaces)  and  assignment  of  functional  responsibility. 

•  Mission  Area  -  Architectures  are  used  at  the  mission  area  level  to  better  manage 
capabilities  within  and  across  mission  areas  and  improve  investment  decisions. 
Architectures  at  this  level  are  federated  to  support  the  development  of  enterprise 
architectures.  They  also  provide  roadmaps  and  descriptions  of  future  or  desired  end 
states. 

•  Component  and  Program  -  Architectures  are  used  at  the  component  and  program  level  to 
identify  capability  requirements  and  operational  resource  needs  that  meet  business  or 
warfighting  objectives.  Component  and  program  architectures  may  then  be  integrated  to 
support  decision  making  at  the  mission  level. 


Besides  these  practical  system  architecture  uses,  architectures  within  the  DoD  are  created  to 
comply  with  law  and  policy.  Tables  4  and  5  describe  the  various  federal  policies  (DoD,  2007a: 
3-2)  and  DoD  directives  (DoD,  2007a:  3-3)  specifying  architecture  use. 

In  response  to  all  these  directives  and  to  aid  the  DoD  in  developing  architectures, 
DoDAF,  volume  I  (DoD,  2007a:  1-1)  quotes  USD(A&T),  ASD(C3I)  and  J6  as  stating  that  “The 
Defense  Science  Board  and  other  major  studies  have  concluded  that  one  of  the  key  means  for 
ensuring  interoperable  and  cost-effective  military  systems  is  to  establish  comprehensive 
architectural  guidance  for  all  of  DoD."  Therefore,  it  is  essential  to  remember  that  good 
architectures  lead  to  good  interoperability.  This  guidance  is  embodied  in  the  DoD  Architecture 
Framework  (DoDAF)  which  currently  is  in  version  1.5. 
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Table  4.  Federal  Policy  for  Architectures  (DoD,  2007a:  3-2) 


Policy/Guidance 

Description 

Clinger-Cohen  Act  of 

1996 

Recognizes  the  need  for  Federal  Agencies  to  improve  the 
way  they  select  and  manage  IT  resources  and  states 
information  technology  architecture,  with  respect  to  an 
executive  agency,  means  an  integrated  framework  for 
evolving  or  maintaining  existing  IT  and  acquiring  new  IT  to 
achieve  the  agency’s  strategic  goals  and  information 
resources  management  goals”.  Chief  Information  Officers 
are  assigned  the  responsibility  for  “developing,  maintaining, 
and  facilitating  the  implementation  of  a  sound  and 
integrated  IT  architecture  for  the  executive  agency.” 

Office  of  Management  and 

Budget  Circular  A- 130 

“Establishes  policy  for  the  management  of  Federal 
information  resources”  and  calls  for  the  use  of  Enterprise 
Architectures  to  support  capital  planning  and  investment 
control  processes.  Includes  implementation  principles  and 
guidelines  for  creating  and  maintaining  Enterprise 
Architectures. 

E-Govemment  Act  of  2002 

Calls  for  the  development  of  Enterprise  Architecture  to  aid 
in  enhancing  the  management  and  promotion  of  electronic 
government  services  and  processes. 

OMB  Federal  Enterprise 
Architecture  Reference  Models 
(FEA  RM) 

Facilitates  cross-agency  analysis  and  the  identification  of 
duplicative  investments,  gaps,  and  opportunities  for 
collaboration  within  and  across  Federal  Agencies. 

Alignment  with  the  reference  models  ensures  that  important 
elements  of  the  FEA  are  described  in  a  common  and 
consistent  way.  The  DoD  Enterprise  Architecture  Reference 
Models  are  aligned  with  the  FEA  RM. 

OMB  Enterprise  Architecture 
Assessment  Framework 
(EAAF) 

Serves  as  the  basis  for  enterprise  architecture  maturity 
assessments.  Compliance  with  the  EAAF  ensures  that 
enterprise  architectures  are  advanced  and  appropriately 
developed  to  improve  the  performance  of  information 
resource  management  and  IT  investment  decision  making. 

General  Accounting  Office 
Enterprise  Architecture 
Management  Maturity 

Framework  (EAMMF) 

“Outlines  the  steps  toward  achieving  a  stable  and  mature 
process  for  managing  the  development,  maintenance,  and 
implementation  of  enterprise  architecture.”  Using  the 
EAMMF  allows  managers  to  determine  what  steps  are 
needed  for  improving  architecture  management. 
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Table  5.  DoD  Decision  Support  Process  (DoD,  2007a:3-3) 


Process 

Description 

Joint  Capabilities  Integration 
and  Development  System 

“Requires  a  collaborative  process  that  utilizes  joint  concepts 
and  integrated  architectures  to  identify  prioritized  capability 
gaps  and  integrated  joint  DOTMLPF  and  policy  approaches 
(materiel  and  non-materiel)  to  resolve  those  gaps.” 

Incorporates  the  requirement  for  the  net-ready  key 
performance  parameter  (NRKPP)  in  accordance  with  DoD 
Directive  4630.5,  DoD  Instruction  4630.8,  and  Chairman 

Joint  Chiefs  of  Staff  (CJCS)  Instruction  (CJCSI)  6212. 01D. 

Planning,  Programming, 
Budgeting,  and  Execution 

DoD  policy  has  not  formalized  the  use  of  architectures  in  the 
PPBE  process  but  DoD  Services,  such  as  the  Navy  and  Air 
Force,  have  noted  that  architectures  provide  a  context  for 
developing  program  priorities,  formulating  programmatic 
modifications,  and  making  IT  investment  decisions. 

Defense  Acquisition  System 

Includes  the  requirement  for  an  integrated  architecture  in 
developing  integrated  plans  or  roadmaps  to  conduct  capability 
assessments,  guide  systems  development,  and  define  the 
associated  investment  plans  as  the  basis  for  aligning 
resources. 

Portfolio  Management 

Calls  for  “the  management  of  selected  groupings  of  IT 
investments  using  strategic  planning,  architectures,  and 
outcome-based  performance  measures  to  achieve  a  mission 
capability”. 

The  actual  act  of  architecting  itself  is  defined  by  Maier  and  Rechtin  (2002: 1)  as  “the  art 
and  science  of  designing  and  building  systems.”  This  is  an  important  recognition  that 
architecting  has  both  a  scientific  approach  and  a  practiced  approach  as  “a  process  of  insights, 
vision,  intuitions,  judgment  calls,  and  even  taste”  (Maier  and  Rechtin  2002:2).  As  such,  many 
different  approaches  may  be  taken  to  developing  architecture  with  differing  emphasis  on  what  is 
important. 
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2.5.2.  Importance  of  the  “ilities” 


As  part  of  the  art  of  architecting,  a  key  aspect  in  determining  the  value  of  a  system  or  its 
architecture  lies  in  an  examination  of  the  “ilities.”  These  are  defined  as  “the  operational  and 
support  requirements  a  program  must  address  (e.g.,  availability,  maintainability,  vulnerability, 
reliability,  supportability,  etc.)”  (Haskins,  2006:Appendix  p6).  As  Dahlgren  and  de  Neufville 
(2007:2)  pointed  out,  “Systems  engineers  need  to  understand  why  successful  systems  perform 
well  in  the  “ilities”  (flexibility,  adaptability,  upgradeability,  reliability,  scalability,  and 
robustness)  and  others  don’t  so  that  they  can  incorporate  that  successful  thought  process  into  the 
design,  development,  and  spiral  development  of  new  systems.”  In  the  March  2003  Software 
Engineering  Institute’s  (SEI)  Workshop  on  the  DoDAF  and  software  architecture,  the 
discussions  point  out  that  “some  parts  of  the  community  believe  that  architecture  is  shaped  more 
by  its  quality  attributes  or  “ilities”  (performance,  availability,  modifiability,  security,  usability, 
etc.)  than  by  its  functionality”  (Wood,  2003: 10).  Voas  (2004: 14)  likens  the  "itities"  to  a  secret 
sauce. 

The  -ilities  (or  software  attributes)  are  a  collection  of  closely  related  behaviors  that  by 
themselves  have  little  or  no  value  to  the  end  users,  but  they  can  greatly  increase  a 
software  application  or  system’s  value  when  added.  To  use  an  analogy,  an  -ility  in  an 
application  or  system  is  like  a  condiment  on  an  entree:  not  valuable  as  a  stand-alone  item 
but  capable  of  significantly  enhancing  the  flavor  when  added  properly. 

Only  a  few  of  the  "ilities"  mentioned  here  are  specifically  identified  in  literature  and  captured  in 

more  detail  in  the  matrix  found  in  Appendix  A.  No  standard  list  of  applicable  "ilities"  exists  as 

almost  any  attribute  imaginable  may  be  implemented  as  an  "ility"  by  just  adding  “ility”  to  the 

end  of  it  as  evidenced  by  the  large  collection  listed  on  wikipedia  (Wikipedia,  2008). 
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2.5.3.  Architecture  Evaluation 

In  the  course  of  actually  examining  these  "ilities"  in  any  attempt  to  determine  their 
quality,  this  process  indeed  falls  into  the  category  of  more  art  than  direct  science.  Continuing  his 
analogy,  Voas  (2004:14)  points  out  the  importance  of  degrees  of  goodness  such  as  putting  just 
enough  or  too  much  salt  on  a  steak  makes  it  either  taste  great  or  be  difficult  to  eat.  As  such, 
directly  measuring  certain  quality  attributes  may  not  be  possible  and  require  nonnumeric  scoring 
techniques.  Others  such  as  Lu  Han  (2006:1),  however,  argue  (albeit  referring  specifically  to 
computing-related  systems  involving  human-factors  considerations)  that  “measuring  ilities  in  a 
general  way  is  hopeless.” 

Several  means  of  evaluating  architectures  or  specific  attributes  exist.  However,  in  the  course  of 
the  authors’  literature  review,  very  little  was  found  in  terms  of  attempting  to  provide  a  quality 
score  related  specifically  to  DoDAF  architectures.  This  gap  in  the  literature  reconfirms  the 
Software  Engineering  Institute's  (2003:10)  finding  that  specific  "analysis  methods  for  the 
DoDAF  have  not  been  reported  publicly."  Most  of  the  existing  methods  such  as  the  Enterprise 
Architecture  Assessment  Framework  (EAAF)  (OMB,  2008),  Enterprise  Architecture 
Management  Maturity  Framework  (EAMMF)  (GAO,  2003),  Interoperability  Score  (i-Score) 
(Ford,  2008),  Multi-Attribute  Tradespace  Exploration  (MATE)  (Ross  and  Hastings,  2006), 
Architecture  Level  Modifiability  Analysis  (ALMA)  (Bengtsson,  2004),  System  Engineering 
Process  Activities  (SEP A)  (Barber,  2003),  International  Standards  Organization/Intemational 
Electrotechnical  Commision  (ISO/IEC)  9126  (Botella,  2004),  Software  Architecture  Analysis 
Method  (SAAM)  (Kazman,  1994),  and  Architecture  Tradeoff  Analysis  MethodSM  (AT AM) 
(Kazman,  2000),  apply  more  to  software  coding  or  were  deemed  inappropriate  for  the  scope  of 


this  effort  in  evaluating  the  architecture  products.  However,  building  on  some  of  these  methods’ 
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concepts,  the  Architecture  Evaluation  Framework  (AEF)  (Lehto,  2005;  Mazhelis,  2006)  and  the 
Enterprise  Architecture  Score  Card™  (Schekkerman,  2004;  Jamison,  2005)  did  provide  relevant 
insight  into  methodologies  more  closely  scoped  to  this  effort.  The  range  in  these  models’  scope 
compared  to  the  target  scope  for  this  effort  is  depicted  in  Figure  9. 


Figure  9.  Architecture  Evaluation  by  Focus  and  Effort 
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2.5.3. 1.  Enterprise  Architecture  Assessment  Framework  (EAAF) 


The  EAAF  is  used  by  the  Office  of  Management  and  Budget  (OMB)  to  evaluate  the 
maturity  and  effectiveness  of  federal  agency  enterprise  architecture  programs.  Specifically,  the 
EAAF  checks  compliance  with  architecture  mandates  such  as  the  Clinger-Cohen  Act  and  OMB 
A- 130.  This  framework  comprises  14  assessment  criteria  where  each  criterion  consists  of  five 
maturity  levels  (OMB,  2008).  This  framework  was  considered  out  of  scope  for  use  in  this 
research  effort  because  of  the  focus  on  higher,  agency-level  compliance  issues  rather  than 
system-related. 

2.5.3.2.  Enterprise  Architecture  Management  Maturity  Framework  (EAMMF) 

Similar  to  the  EAAF,  the  EAMMF  is  used  by  the  General  Accounting  Office  (GAO)  to 
evaluate  maturity  and  the  steps  needed  to  improve  architecture  management.  Comprised  of  3 1 
core  elements,  5  stages,  and  4  attributes,  the  EAMMF  is  also  a  means  for  checking  agency 
compliance  with  federal  policy  (GAO,  2003).  This  framework  was  considered  out  of  scope  at 
the  same  level  as  the  EAAF. 

2.5.3.3.  Interoperability  Score  (i-Score) 

While  i-Score  only  addresses  a  single  aspect  of  the  overall  system  and  architecture,  it  was 
important  to  review  Ford’s  (2008)  work  for  an  understanding  of  the  possible  depth  and 
quantifiability  one  could  go  into  in  determining  each  specific  area  of  interest’s  measure  of 
quality.  With  the  drive  toward  network-centric  operations,  an  increased  focus  of  research  has 
tried  to  improve  the  interoperability  of  systems.  Ford’s  (2008:2)  research  presents  the  i-Score  as 
“a  generalized  measure  of  the  interoperability  of  systems  of  all  types,  supporting  an  operational 
thread.” 
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The  i-Score  methodology  examines  “existing  architecture  data  (specifically,  DoDAF  OV- 
5,  OV-2,  and  SV-3)  and  applies  graph,  optimization  and  interoperability  theory  to  provide  a 
generalized  measurement  of  interoperability”  (Ford,  2008:2).  The  methodology  walks  through 
the  six  steps  of  (Ford,  2008:3-5): 

1)  diagram  the  operation  thread  and  define  the  set  of  supporting  systems; 

2)  create  an  interoperability  matrix; 

3)  calculate  the  i-Score; 

4)  determine  the  optimum  i-Score; 

5)  calculate  the  interoperability  gap;  and 

6)  perform  interoperability  analysis. 

This  method  results  in  a  single  number  measure  between  zero  and  one  of  how  well  the  system 
interoperates  along  the  examined  operational  thread  (Ford,  2008:4).  While  this  groundbreaking 
research  provides  a  quantifiable  interoperability  number,  this  thesis's  authors  determined  the 
depth  of  analysis  to  reach  this  number  was  significantly  deeper  than  any  other  measures  and  the 
intent  of  the  VDEA  scorecard.  i-Score  may  prove  useful,  however,  if  future  research  in  other 
value  measures  enables  a  similar  depth  of  analysis.  VDEA  could  be  the  framework  that  binds 
such  measures  together. 

2.5.3.4.  Multi- Attribute  Tradespace  Exploration  (MATE) 

While  not  necessarily  focused  on  evaluating  the  quality  of  system  architecture,  MATE 
provides  additional  insight  to  the  importance  of  architectures  and  means  of  making  tradeoff 
decisions  based  on  the  architecture.  MATE  began  as  a  process  to  incorporate  decision  theory 
into  model  and  simulation-based  design  primarily  applied  to  the  space  domain  (Ross,  2003:3). 


35 


Through  numerous  research  efforts  at  the  Massachusetts  Institute  of  Technology's  Systems 
Engineering  Advancement  Research  Initiative  (MIT  SEARi);  MATE  continues  to  evolve  and 
find  new  areas  of  application  such  as  additions  for  systems  of  systems  design  (Chattopadhyay, 
2008),  value  robustness  (Ross,  2008),  providing  a  framework  for  incorporating  "ilities"  into 
tradespace  studies  (McManus,  2006),  and  quantifying  important  system  "ilities"  such  as 
flexibility,  survivability,  and  changeability  (Ross,  2006).  While  finding  numerous  applications 
to  address  such  attributes  as  changeability,  survivability,  flexibility,  robustness,  and  other  ilities, 
the  more  detailed  level  of  MATE  analysis  and  its  application  more  to  system  characteristics  than 
the  architecture  themselves  is  beyond  the  scope  of  this  effort. 

2.5.3.5.  Architecture  Level  Modifiability  Analysis  (ALMA) 

This  method  focuses  more  narrowly  on  the  analysis  of  how  modifiable  the  architecture  is 
and  specifically  focused  on  software  architectures.  As  reported  by  Bengtsson  (2004),  ALMA 
was  the  combination  of  independent  work  by  Bengtssom  and  Bosch  (1999)for  predicting 
maintenance  efforts  based  on  the  system's  software  architecture  as  well  as  the  work  of  Lassing  et 
al.  (1999)  for  identifying  inflexibility  at  the  software  architecture  level.  ALMA  uses  a  "unified 
architecture-level  modifiability  analysis  method  that;  distinguishes  multiple  analysis  goals,  has 
visible  assumptions  and  provides  repeatable  techniques  for  performing  the  steps"  (Bengtsson, 
2004:129-130). 

The  five  main  steps  of  ALMA  are  selecting  the  goal,  describing  the  software  architecture, 
developing  the  scenario,  and  evaluating  and  interpreting  the  scenario.  Different  specific 
techniques  are  used  in  some  of  these  main  steps  depending  on  the  goal.  In  general,  the  goal  is 
typically  one  of  the  following  three:  "prediction  of  future  maintenance  cost,  identification  of 
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system  inflexibility  and  comparison  of  two  or  more  alternative  architectures"  (Bengtsson, 

2004: 130).  This  method's  modifiability  analysis  method  was  determined  too  narrow  for  the 
thesis  problem. 

2.5.3.6.  System  Engineering  Process  Activities  (SEPA) 

Another  method  reviewed  for  evaluating  architectures  is  SEPA.  While  not  dealing  with 
quality  attributes  directly,  SEPA  focuses  on  requirements  and  architecture  in  the  software  realm. 
SEPA's  objective  is  "to  enable  comprehensive  support  for  architecture  derivation  and  evaluation 
through  formal  processes  and  complementary  tools  emphasizing  architecture  analysis  as  well  as 
requirements  management"  (Barber,  2003:1).  SEPA  emphasizes  early  evaluation  of  the 
architecture  in  the  development  process.  The  intent  of  this  evaluation  is  to  provide  an  early 
opportunity  to  fix  requirements  errors  as  well  as  ensure  the  software  architecture's  accuracy  for 
use  in  building  the  system.  This  method  utilizes  a  number  of  tools,  models,  and  simulations 
inappropriate  for  this  thesis  problem. 

2.5.3.7.  ISO/IEC  9124  (Botella,  2004) 

As  one  of  the  most  widespread  quality  models,  the  International  Standards  Organization's 
ISO/IEC  9124  serves  as  a  guide  for  the  evaluation  of  software  quality  which  defines  a  general 
quality  model  framework  applicable  to  different  kinds  of  software.  Most  importantly,  ISO/IEC 
9124  defines  six  higher-level  product  quality  characteristics  which  are  divided  into  other  sub¬ 
characteristics  as  shown  in  Table  6  and  are  then  decomposed  into  attributes  producing  a 
multilevel  hierarchy.  The  attributes  at  the  bottom  of  the  hierarchy  should  be  measureable 
software  attributes  which  can  have  a  quality  value  determined  by  applying  some  metric.  While 
generic  in  nature  and  specifically  geared  towards  software,  ISO/IEC  9124  still  provides  more 
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guidelines  for  the  consideration  of  quality  values  which  may  apply  to  a  more  generic  system 
architecture  value  hierarchy. 


Table  6.  ISO  Values  (Botella,  2004) 


Characteristics 

Sub-characteristics 

Functionality 

Suitability 

Accuracy 

Interoperability 

Security 

Functionality  Compliance 

Reliability 

Maturity 

Fault  Tolerance 

Recoverability 

Reliability  Compliance 

Usability 

Understandability 

Leamability 

Operability 

Attractiveness 

Usability  Compliance 

Efficiency 

Time  Behavior 

Resource  Utilization 

Efficiency  Compliance 

Maintainability 

Analysability 

Changeability 

Stability 

Testability 

Maintainability  Compliance 

Portability 

Adaptability 

Installability 

Co-existence 

Replaceability 

Portability  Compliance 

38 


2.5.3.8.  Software  Architecture  Analysis  Method  (SAAM) 


As  its  name  implies,  SAAM  is  likewise  focused  on  software  systems  and  the  stakeholder. 
In  regards  to  this  effort,  the  focus  on  software  architecture  is  too  narrow  to  directly  apply  to  this 
thesis  effort.  SAAM  specifies  functionality,  structure,  and  allocation  as  three  important 
"perspectives  for  understanding  and  describing  architectures"  (Kazman,  1994).  SAAM  has  also 
been  extended  to  assess  software  architectures  with  respect  to  different  quality  factors  by 
obtaining  scenarios  from  the  stakeholders  and  then  exploring  their  effects  on  the  architecture.  In 
particular,  much  work  has  focused  on  architectural  analysis  of  the  individual  attributes  of 
modifiability,  performance  analysis,  availability  analysis,  and  security  analysis.  The  SAAM 
process  consists  of  the  four  major  steps  of  developing  scenarios,  describing  the  architectures, 
evaluating  the  scenarios  and  performing  an  overall  evaluation  (Kazman,  1994). 

2.5.3.9.  Architecture  Trade-off  Analysis  Method  (ATAMsm) 

Growing  on  the  work  from  SAAM,  the  AT  AM  is  developed  for  the  architecture  of 
complex  software  intensive  systems  as  "a  method  for  evaluating  architecture-level  designs  that 
considers  multiple  quality  attributes"  (Kazman,  1998:1).  The  goal  is  to  gain  early  insight  into 
whether  or  not  the  complete  architecture  meets  requirements.  While  also  more  narrow  and 
detailed  to  apply  directly,  ATAM  provides  some  useful  concepts  to  consider. 

Where  other  methods  focus  on  individual  attributes,  ATAM  attempts  to  capture  the 
impact  of  interactions  between  attributes.  This  method  intends  to  find  trade-off  points  between 
attributes,  improve  communication  between  stakeholders  with  regard  to  each  attribute,  clarify 
and  refine  the  requirements,  and  provide  the  necessary  framework  for  ongoing,  simultaneous 
system  design  and  analysis  processes.  The  four  main  areas  of  effort  comprise  the  ATAM  are: 
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"scenario  and  requirements  gathering,  architectural  views  and  scenario  realization,  model 
building  and  analysis,  and  tradeoffs"  (Kazman,  1998:2). 

2.5.3.10.  Architecture  Evaluation  Framework  (AEF) 

Building  on  the  ATAM  concepts,  the  AEF  was  developed  to  define  the  necessary  tools 
and  procedures  to  evaluate  system  architecture  within  the  telecommunications  domain.  The  first 
step  is  creating  a  hierarchy  with  more  generic  top-level  factors  based  on  their  identified  relevant 
business  drivers  down  to  more  specific  leaf-level  factors.  Next,  their  relative  importance  is 
determined  by  applying  weights  according  the  Analytic  Hierarchy  Process  (AHP)  technique. 
Here,  a  pair-wise  comparison  of  each  branch  and  each  level  is  conducted  in  relation  to  a  specific 
business  driver.  For  each  of  the  lower  leaf-level  factors,  measures  are  created  in  the  form  of 
specific  questions.  The  evaluation  team  then  answers  each  question  to  evaluate  the  effect  that 
answer  has  on  the  specific  business  driver  being  considered.  This  effect  is  then  scored  as  a 
number  on  a  scale  of  zero  (has  a  negative  effect)  to  one  (has  a  positive  effect).  These  values, 
when  combined  with  their  relative  weighting,  "are  used  to  evaluate  the  overall  appropriateness 
score  of  the  architecture"  (Mazhelis,  2006:3).  Alternative  architectures  can  subsequently  be 
compared  as  well.  Likewise,  a  sensitivity  analysis  can  be  made  to  evaluate  changes  in  the  score 
due  to  changes  in  the  weights  (Lehto,  2005;  Mazhelis,  2006). 

While  the  AEF  was  tailored  to  the  company  and  their  specific  needs,  this  method's 
approach  provides  a  close  comparison  at  a  high  level  to  the  VFT  approach.  However,  the  use  of 
the  AHP  technique  is  a  notable  exception.  Considered  overly  complex  for  this  thesis  effort,  AHP 
often  requires  "extensive  pair-wise  comparisons...  and  extensive  mathematical 
calculations... [which]  seem  to  obscure,  rather  than  illuminate,  the  tradeoffs"  (Kirkwood, 
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1997:260).  Additionally,  adding  a  new  value  to  the  mix  would  require  a  potentially  lengthy 
recalculation  of  the  pair-wise  comparisons. 

2.5.3.11.  Enterprise  Architecture  Score  Card™ 

One  of  the  methods  discovered  closer  in  scope  to  provide  a  high-level  measure  of 
architecture  quality  and  completeness  is  the  Enterprise  Architecture  Score  Card™  developed  by 
Schekkerman  (2004).  EAS  is  geared  more  to  industry’s  approach  to  architecture  versus  the  DoD 
with  its  greater  emphasis  on  business  drivers.  While  considered  too  qualitative  for  this  research 
effort,  EAS  helps  distinguish  an  upper  bound  for  the  level  of  detail  focus  for  the  direction  of  this 
research  effort. 

EAS’s  goal  is  “to  help  understand  the  relations  and  elements  that  influence  the  decision¬ 
making  about  the  adoption  of  enterprise  architecture  concepts  in  several  ways”  (Schekkerman, 
2004:3).  It  further  serves  to  communicate  “the  essential  elements  and  functioning  of  the 
enterprise”  (Schekkerman,  2004:3)  by  providing  a  three  point  score  (0-unclear,  1 -partially  clear, 
2-clear)  highlighting  areas  that  are  good  or  need  further  development.  The  Extended  Enterprise 
Architecture  Framework  (E2AF)™  forms  the  basis  of  the  scorecard  in  a  matrix  of  four  aspect 
areas  and  six  abstract  levels  of  concern. 

The  four  aspect  areas  are  Business,  Information,  Information  Systems,  and  Technology 
Infrastructure.  The  Business  aspect  is  the  starting  point  involving  the  organizational  and 
management  processes  in  the  architecture.  The  Information  aspect  is  extracted  from  the  business 
aspect  to  express  the  information  needs;  flows  and  relationships  help  to  identify  which  functions 
can  be  automated.  Information  Systems  then  covers  that  automated  support,  while  Technology 
Infrastructure  covers  the  supporting  technology  environment  for  the  information  systems. 
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The  six  abstract  levels  of  concern  are  Contextual,  Environmental,  Conceptual,  Logical, 
Physical,  and  Transformational.  The  Contextual  level  (“Why?”)  describes  the  mission,  vision, 
and  scope  of  the  organization  and  architecture.  The  Environmental  level  (“With  Who?”) 
examines  the  extended  business  relationships  and  information  flows.  The  Conceptual  level 
(“What?”)  focuses  on  the  goals,  objectives,  and  requirements  of  the  entities  involved.  The 
Logical  level  (“How?”)  explores  the  ideal  logical  solutions.  The  Physical  level  (“With  What?”) 
addresses  the  physical  solutions  and  supporting  products.  Finally,  the  Transformational  level 
(“When?”)  describes  the  proposed  solutions’  impacts. 

The  Enterprise  Architecture  Score  Card  methodology  then  builds  on  the  E2AF  by  asking 
questions  at  each  aspect  area  and  abstraction  level.  The  zero  to  one  range  of  answers  to  each 
question  helps  identify  where  the  architecture  fulfills  its  purpose  and  what  areas  need 
improvement.  The  EAS  further  assesses  a  zero  to  one  range  for  integration  to  address  the 
consistency  of  the  architecture.  Finally,  it  is  important  to  not  misinterpret  the  numerical  results 
from  the  EAS.  These  numbers  merely  show  areas  of  strength  and  areas  in  need  of  improvement. 
There  is  no  score  that  specifically  represents  “good”  or  “fail.” 
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III.  Methodology 


As  discussed  previously,  joint  force  protection  faces  numerous  challenges  in  its  net- 
centric  transformation  especially  in  interoperability.  A  key  enabler  to  good  interoperability  is  a 
good  architecture.  The  Security  Equipment  Integration  Working  Group  (SEIWG),  within  the 
aspect  of  their  mission  to  coordinate  and  influence  system  architecture,  desired  a  tool  to  evaluate 
the  quality  of  their  proposed  "to-be"  architecture.  As  described  in  the  previous  chapter,  the 
architecture  evaluation  tools  fell  short  of  the  desired  capability.  Therefore,  the  principles  of 
Value-Focused  Thinking  (VFT)  also  described  in  the  previous  chapter  guided  the  development 
of  a  new  tool— the  Value-Driven  Enterprise  Architecture  Score  (VDEA-Score). 

This  chapter  describes  the  methodology  used  to  identify  the  problem  and  develop  the 
weighted  hierarchy  with  measures  and  value  functions  of  the  values  deemed  most  important  to 
the  stakeholder.  This  forms  the  VDEA-Score  model  for  evaluation.  For  the  purpose  of  this 
thesis,  the  emphasis  is  on  the  architecture  quality  values  meaning  the  intrinsic  quality  of  the 
products  themselves  in  terms  of  documentation  standards  and  desired  attributes.  The  alternative 
generation  and  scoring  process  will  also  be  discussed.  Finally,  discussion  of  the  applicability  of 
the  VDEA-Score  model  of  architecture  quality  values  to  another  system's  architecture  concludes 
this  chapter. 

3.1.  Problem  Identification 

For  this  effort,  the  core  question  asked  by  the  decision  maker  was  to  determine  if 
common  joint  force  protection  values  could  be  used  as  a  basis  for  evaluating  a  “To-Be” 
architecture  for  net-centric  force  protection.  The  research  team  answered  this  question  by 
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creating  a  new  VDEA-Score  evaluation  methodology  used  to  develop  a  single  joint  force 
protection  value  model  with  measures  of  effectiveness  and  evaluate  the  candidate  joint  force 
protection  architecture.  This  value  model  may  aid  in  future  evaluating,  scoring,  and  ranking  “To- 
Be”  architectures  based  on  values  important  to  the  decision  maker.  This  VDEA-Score  allows  the 
decision  maker  to  measure  the  effects  of  changes  to  Concept  of  Operations  (CONOPS), 
resources,  or  level  of  net-centricity  as  proposed  in  revisions  of  the  “To-Be”  architectural  product 
suite  and  determine  the  degree  of  change  to  the  overall  value  expected  to  joint  force  protection. 

A  weighted,  hierarchical  tree  of  component  values  of  the  Joint  Force  Protection 
architecture  was  thus  desired  to  identify  components  that  are  influenced  by  net-centricity  and 
interoperability.  In  addition,  the  decision  maker  wanted  a  set  of  measures,  with  associated  utility 
curves,  to  evaluate  the  degree  to  which  each  value  component  was  achieved  within  a  DoDAF 
architectural  product  suite.  Lastly,  the  decision  maker  wanted  a  composite  value-focused  Joint 
Force  Protection  score  for  an  overall  CONOPS  as  depicted  in  a  suite  of  architectural  products. 
This  would  create  a  single  measure  for  the  value  created  by  investing  Joint  Force  Protection 
resources  to  match  the  “To-Be”  architecture.  Therefore,  the  problem  was,  “How  should  common 
Joint  Force  Protection  values  be  used  to  evaluate  a  “To-Be”  architecture  for  net-centric  force 
protection?” 

3.2.  Develop  and  Verify  Value  Hierarchy 

The  initial  value  hierarchy  was  formed  by  two  branches  divided  into  an  architecture- 
specific  branch  and  a  system-specific  branch.  This  approach  enhanced  the  hierarchy’s 
decomposability  by  dividing  it  into  an  architecture-specific  branch  to  address  the  quality  of  the 
architectural  views  or  products  and  a  system-specific  branch  to  address  the  effectiveness  quality 
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of  the  system  represented  by  the  architectural  views  or  products.  The  two-branch  division  also 
maintains  exclusivity  of  component  value  between  the  architecture  quality  and  system 
effectiveness  values  which  allows  for  full  separation  of  the  two  branches  for  separate  reuse 
across  diverse  applications  supporting  Kirkwood’s  (1997)  desirable  property  of  nonredundancy. 
This  division  further  supports  Kirkwood’s  (1997)  other  desirable  property  of  easier  operability. 
Not  only  is  the  hierarchy  easier  to  read,  but  the  two-branch  division  also  facilitates  reuse 
especially  of  the  architecture  quality  values  to  apply  to  another  program’s  architecture. 

To  develop  an  initial  set  of  “ility”  values,  a  number  of  questions  were  considered  by  the 
authors  based  on  personal  experience  and  literature  review  such  as:  What  are  the  overall 
objectives?  What  values  are  essential  to  ensuring  effective  joint  force  protection?  What  values 
are  essential  to  architectures?  As  discussed  in  Chapter  2,  no  standard  list  of  applicable  “ilities” 
exists.  The  table  in  Appendix  A  represents  the  comprehensive  list  of  possible  values  compiled 
by  the  authors  through  the  literature  review  (e.g.,  Bottella  (2004),  Lehto  (2005),  Ross  (2006), 
Dalgren  (2007),  and  others)  and  brainstorming  sessions.  The  Wikipedia  (2008)  listing  under 
“ilities”  was  also  included  as  considered  by  the  authors  as  an  internet  brainstorming  product. 

Using  the  affinity  diagram  technique,  the  large  list  of  "ilities"  was  converted  to  individual 
note  cards.  The  research  team  physically  arranged  the  cards  without  discussion  into  stacks  of 
related  terms  resulting  in  30  different  groupings.  After  this  initial  grouping,  discussion  ensued 
amongst  the  team  which  further  refined  the  groupings. 

As  part  of  this  discussion,  while  keeping  Kirkwood’s  (1997)  principles  of  small  size  and 
completeness  in  mind,  a  number  of  subgroups  and  individual  attributes  were  discarded  as  not 
applicable  to  this  effort.  The  remaining  22  subgroups  were  examined  for  consolidation  because 
some  attributes  could  be  considered  synonyms  or  within  the  definitional  scope  of  others. 
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Turning  these  subgroups  into  values  for  the  hierarchy,  the  "ility"  with  the  widest  definitional 
scope  was  chosen  and  defined  such  that  it  could  be  decomposed  by  the  other  attributes  in  the 
subgroup.  Likewise,  the  other  "ilities"  in  the  respective  subgroups  were  defined  to  cover  the 
important  values  in  as  few  or  decomposable  attributes  as  possible.  The  resulting  set  of  two 
complete,  main  groups  emerged  as  the  Architecture  Quality  Values  and  System  Effectiveness 
Values.  These  two  main  groups  formed  the  branches  with  subsequent  tiers  formed  with  their 
associated  subgroups  and  attributes  to  establish  the  initial  value  hierarchy. 

Using  the  initial  value  hierarchy  as  a  starting  point,  the  decision  maker  was  interviewed 
to  raise  discussion  and  educe  important  values  that  the  authors  may  have  initially  overlooked. 

The  resulting  value  hierarchy  established  during  this  interview  process  is  exhibited  in  Figure  10 
with  the  additional  tiers  shown  in  Figure  1 1  for  the  System  Effectiveness  Values  and  Figure  12 
for  the  Architecture  Quality  Values.  The  decision  maker  agreed  that  the  proposed  value 
hierarchy  accurately  mirrored  values  essential  to  this  project.  The  second-tier  objectives  are 
general  values  essential  to  first-tier  branches.  The  third-tier  values  are  supporting  values  that 
provide  greater  detail  about  what  is  meant  by  the  general  second-tier  value  and  so  forth.  The 
resulting  hierarchy  also  satisfies  Kirkwood’s  (1997)  principles  of  completeness,  non-redundancy, 
decomposability,  operability,  and  (relatively)  small  size. 

3.2.1.  System  Effectiveness  Value 

For  this  effort,  System  Effectiveness  was  defined  as  "the  quality  of  the  instantiated 
system  being  represented  and  its  ability  to  perform  its  stated  mission."  While  the  authors  believe 
these  values  of  Capability,  Maintainability,  and  Interoperability  are  applicable  to  most  DoD 
systems  at  a  high  level,  they  were  specifically  defined  to  the  force  protection  domain  through 
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their  lower-tier  values  (Mills,  2009).  The  System  Effectiveness  Value  branch  of  the  hierarchy  is 
provided  in  Figure  1 1  for  reference  because  the  focus  of  this  paper  is  on  the  Architecture  Quality 
Values. 

3.2.2.  Architecture  Quality  Values 

This  branch,  shown  in  Figure  12,  was  defined  as  "the  intrinsic  quality  of  the  products  in 
terms  of  documentation  standards  and  desired  attributes."  The  authors  contend  that  the  values 
contained  therein  are  applicable  to  any  DoDAF  architecture,  independent  of  the  described 
system.  Table  7  expands  the  definition  of  each  value  in  the  Architecture  Quality  (AQ)  sub-tier. 
Similar  information  can  be  found  in  Mills  (2009)  for  the  System  Effectiveness  (SE)  sub-tier 
values.  The  asterisk  notes  a  net-centric  relation. 


Joint  Force  Protection  VDEA- 
Score 


System  Effectiveness 
Value 


Architecture  Quality 
Value 


Figure  10.  VDEA-Score  Hierarchy  with  First-Tier  Branch 
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Figure  11.  System  Effectiveness  Values  Branch 
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Figure  12.  Architecture  Quality  Value  Branch 
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Table  7.  Architecture  Quality  Value  Definitions 


Accessibility 

The  assurance  that  information  relating  to  architecture  products  can 
only  be  accessed  or  modified  by  those  authorized  to  do  so,  preventing 
information  use  outside  the  architecture’s  intended  context. 

Subscribability* 

How  easily  the  information  pertinent  to  a  stakeholder  can  be 
accessed. 

Controllability* 

The  assurance  that  only  those  authorized  to  modify  architecture 
information  can  do  so  with  appropriate  revision  control  measures. 

Protectability* 

The  assurance  that  only  those  authorized  to  access  the  information 
may  do  so. 

Usability 

The  extent  to  which  the  architecture  framework  can  be  used  by  users 
to  achieve  goals  effectively  and  efficiently. 

Longevity 

The  degree  to  which  the  architecture  product  is  available  over  time 
(i.e.:  documentation). 

Understandability 

The  level  of  difficulty  needed  to  understand  what  the  architecture  is 
conveying. 

Simplicity 

How  many  diverse  and  autonomous,  but  interrelated  and 
interdependent  components  or  parts,  are  linked  through  many 
interconnections. 

Readability 

How  easily  the  information  is  conveyed  to  the  reader. 

Modifiability 

How  easily  the  architecture  framework  can  be  updated,  upgraded,  or 
otherwise  accepts  changes. 

Scalability* 

The  ability  of  the  architecture  to  maintain  its  function  and  retain  its 
desired  properties  when  its  scale  is  increased  greatly  without  having  a 
corresponding  increase  in  complexity. 

Tailorability 

The  ability  of  the  architecture  products’  level  of  detail  to  be  changed 
to  meet  the  needs  of  different  stakeholders. 

Evolvability* 

The  ability  of  the  architecture  to  change  as  needed  to  handle 
refinements. 

Accountability 

The  ability  of  the  architecture  to  be  responsible  for  addressing  the 
stakeholders  requirements. 

Compliancy* 

How  effective  architecture  products  comply  with  DoDAF  standards. 

Traceability 

The  extent  to  which  the  information  in  the  Operational  Views  match 
the  information  in  the  System  Views. 

Consistency 

The  agreement  of  parts  or  features  of  architecture  products  to  one 
another  or  a  whole. 

SME  Input 

The  extent  of  pertinent  Subject  Matter  Expert  involvement  in 
architecture  development 

*  Denotes  net-centric  relationship 
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3.4.  Develop  Evaluation  Measures 


With  the  value  hierarchy  established,  evaluation  measures  for  each  of  the  values  in  the 
last  tier  in  the  hierarchy  were  developed  for  the  evaluation.  A  brief  description  of  each 
Architecture  Quality  Value  evaluation  measure  follows.  These  measures  were  created  in 
consultation  with  and  validated  by  the  decision  maker.  These  measures  are  measurable, 
operational,  and  understandable,  satisfying  Keeney’s  (1992)  three  principles  for  evaluation 
measures.  While  suggested  sources  for  the  evaluator  to  review  in  answering  each  measure  are 
provided,  it  is  important  to  note  that  an  answer  may  also  be  found  through  the  review  of  other 
products.  Appendix  B  serves  as  a  summary  evaluation  sheet  organized  by  value  with  each 
measure  name,  the  respective  evaluation  question,  and  the  possible  result.  Data  collected  for 
each  evaluation  measure  is  presented  in  Chapter  4. 

3.4.1.  Evaluation  Measures  for  Subscribability 

Two  measures  are  used  to  evaluate  the  Subscribability  of  the  architecture  products.  DoD 
Directive  8320  (2007)  states  data  is  an  essential  enabler  of  net-centric  warfare.  Data  shall  be 
made  visible,  accessible,  and  understandable  for  interoperability  purposes. 

3.4.1. 1.  Access 

The  natural,  direct  ACCESS  measurement  determines  the  degree  of  difficulty  the 
stakeholders  have  in  obtaining  electronic  access  to  the  products.  The  assumption  was  made  that 
all  stakeholders  know  they  are  indeed  stakeholders  and  thus  aware  of  the  existence  of  the 
products  and  the  starting  point  to  obtain  them.  The  AV-1  is  the  best  source  for  describing  the 
process  to  obtain  the  products.  Most  likely,  the  products  are  found  in  an  on-line  repository.  It  is 
possible  the  AY-2  may  also  be  a  source  as  the  repository  may  include  this  information  in  its 
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definition.  If  this  information  cannot  be  found  in  the  architecture  products,  the  evaluator’s 

experience  with  the  repository  may  be  considered.  For  example,  use  of  an  official  DoD  or 

service-level  repository  such  as  the  DoD  Architecture  Repository  System  (DARS)  (DoD,  2009) 

or  the  Air  Force  Architecture  Repository  (Department  of  the  Air  Force,  2009)  assumes  existing 

access  so  the  highest  category  (see  below)  is  scored  because  this  is  not  an  evaluation  of  the 

official,  central  repository  itself.  If  no  share  site  or  repository  is  used,  thus  requiring  point-to- 

point  transfer  (e.g.,  a  stakeholder  has  to  request  email  distribution),  the  lowest  category  is  scored. 

The  possible  score  categories  are: 

o  No  means  to  gain  access 
o  1  week  to  gain  access 
o  3  days  <  access  granted  <  1  week 
o  5  minutes  <  access  granted  <  3  days 
o  Access  granted  <  5  minutes 


3.4.I.2.  Product  Locatability 

The  natural,  direct  PRODUCT  LOCATABILITY  measurement  assesses  the  degree  of 

difficulty  the  stakeholders  have  locating  the  desired  architecture  products  after  access  has  been 

obtained.  The  AV-1  or  AV-2  may  be  sources  for  describing  the  process  for  locating  the 

products.  As  in  the  previous  measure,  the  evaluator’s  experience  with  the  repository  may  be 

considered  if  the  data  structure  is  not  documented  in  the  products.  Likewise,  the  use  of  an 

official  repository  (e.g.,  DARS)  would  score  the  highest  category  while  emailing  products  to 

stakeholders  would  score  the  lowest  category.  The  possible  score  categories  are: 

o  Cannot  locate  products 
o  >5  minutes  to  locate  products 
o  <5  minutes  to  locate  products 
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3.4.2.  Evaluation  Measure  for  Protectability:  Access  Control 


The  ACCESS  CONTROL  measurement  evaluates  the  degree  of  protection  over  the 

architecture  products.  This  constructed,  proxy  measurement  evaluates  the  information  assurance 

issues  of  whether  or  not  access  control  measures  have  been  implemented  appropriately  to  the 

level  of  protection  required.  Note  that  this  assumes  the  architecture  products'  level  of  protection 

is  accurately  described.  For  example,  the  products  posted  to  a  community  site  have  strong  user 

identification  and  password  requirements  to  access.  The  AV-1  or  possibly  TV-1  may  be 

document  sources  to  find  information  related  to  this  measure.  If  not  documented,  the  evaluator 

may  consider  the  protection  provided  by  the  repository.  For  example,  products  located  in  DARS 

by  default  meet  the  highest  category.  A  program-specific  share  site  with  no  protections  for 

official  use  only  documents  would  fall  in  the  lowest  category.  The  possible  score  categories  are: 

o  No  plan  or  inadequate  plan 
o  Plan  exists  but  not  implemented 
o  Appropriate  protection  implemented 

3.4.3.  Evaluation  Measure  for  Controllability:  Document  Protection 

The  DOCUMENT  PROTECTION  measurement  evaluates  the  controllability  of  the 
architecture  products.  This  natural,  direct  measurement  concerns  configuration  control  by 
evaluating  the  degree  of  control  over  the  architecture  products  to  protect  against  unauthorized 
changes.  This  measure  refers  to  the  final  published  products  which  should  be  write-protected. 
Therefore,  an  unauthorized  person  should  not  be  able  to  change  and  republish  the  products  to  the 
repository.  The  AV-1  may  discuss  this  aspect  either  directly,  refers  to  a  configuration  control 
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plan,  or  by  association  through  stating  the  use  of  an  official  repository  as  the  location  for  the 

final  published  products.  If  not  documented,  again,  the  use  of  an  official  common  repository 

meets  the  intent  of  the  highest  category.  If  a  program-specific  repository  is  used,  the  evaluator 

should  examine  the  write  protection  of  the  documents.  The  possible  score  categories  are: 

o  No  plan  for  write  protection 
o  Plan  exists  but  not  implemented 
o  All  products  controlled 

3.4.4.  Evaluation  Measures  for  Longevity 

This  value  consists  of  two  measures  to  ascertain  whether  or  not  the  architecture 
documentation  may  be  available  for  reference  or  reuse  over  an  extended  period  of  time. 

3.4.4.I.  File  Management 

The  constructed,  direct  FILE  MANAGEMENT  measure  examines  the  status  of  an  official 

file  management  system  for  holding  the  architecture  products.  If  one  exists,  it  is  examined  to 

determine  its  effectiveness  by  the  extent  to  which  documents  are  contained  and  maintained 

within  it.  The  AV-1  may  discuss  this  aspect  either  directly  or  by  association  through  stating  the 

use  of  an  official  repository  as  the  location  for  the  final  published  products.  If  not  documented, 

the  use  of  an  official  common  repository  meets  the  intent  of  the  highest  category.  If  a  program- 

specific  repository  is  used,  the  evaluator  should  examine  its  file  structure  and  make  a  judgment 

call  to  determine  if  it  meets  the  intent  of  a  managed  system.  For  example,  if  multiple  and 

differing  versions  of  a  view  are  found  in  different  folders  without  a  naming  convention  to 

identify  them  as  drafts  versus  final,  then  no  credit  for  a  system  should  be  given.  The  possible 

score  categories  are: 

o  No  official  file  management  system 
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o  File  management  system  exists  but  does  not  contain  all  developed  products  or 
products  not  maintained 

o  File  management  system  exists  containing  all  developed  products  and  maintained 
for  currency 


3.4.4.2.  File  Format 

The  constructed,  proxy  FILE  FORMAT  measurement  evaluates  the  degree  to  which 
electronic  copies  of  the  products  are  available  in  an  industry  standard  or  interchangeable  format 
allowing  viewing  over  a  period  of  time.  The  AV-1  is  the  likely  source  regarding  the  tools  used 
which  therefore  drives  the  format  available  for  the  products.  If  this  is  not  documented,  the 
format  of  the  products  reviewed  may  be  evaluated.  The  possible  score  categories  are: 
o  No  electronic  products  or  no  longer  accessible 

o  Proprietary  fde  format  (i.e.  only  accessible  with  one  type  of  proprietary  software) 
o  General  file  format  (i.e.  available  to  common  viewer  such  as  Adobe  Acrobat 
Reader,  OpenOffice.org,  common  web  browser,  etc.) 


3.4.5.  Evaluation  Measure  for  Simplicity 

This  value  consists  of  three  measures  to  ascertain  the  level  of  simplicity  in  the 
architecture  documentation. 

3.4.5.I.  Connections 

The  constructed,  proxy  CONNECTIONS  measurement  examines  how  easy  the  links 
between  entities  are  to  understand.  The  evaluator  examines  the  interfaces  between  steps,  entities, 
activities,  etc.,  of  all  available  products.  A  subjective  determination  is  then  made  if  these  items 
make  sense  or  are  laid  out  in  an  organized  fashion  within  each  available  product.  A  percentage 
is  then  determined  by  the  ratio  of  the  total  number  of  products  in  compliance  to  the  total  number 
of  existing  products. 
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3.4.5.2.  Architecture  Redundancy 


The  natural,  proxy  ARCHITECTURE  REDUNDANCY  measurement  looks  for  any 

unnecessary  duplication  of  information  across  all  available  products.  For  example,  are  there  any 

extra  entities,  activities,  links,  etc.,  unnecessarily  accomplishing  the  same  goal?  Note  that  this 

redundancy  does  not  refer  to  intentionally  designed  redundant  systems.  The  measurement 

categories  are  based  on  one  redundancy  discovered  per  number  of  entities  reviewed.  The 

possible  score  categories  are: 

o  >  1  unnecessary  duplication  per  10  items 
o  1  unnecessary  duplication  between  10  and  100  items 
o  1  unnecessary  duplication  between  100  and  500  items 
o  1  unnecessary  duplication  >  500 

3.4.5.3.  Architecture  Economy 

The  constructed,  proxy  ARCHITECTURE  ECONOMY  measurement  checks  all  available 
products  for  whether  or  not  multiple  steps  are  being  used  unnecessarily  to  represent  the  same 
activity  (e.g.,  could  three  activities  be  represented  sufficiently  by  consolidating  into  one?). 
However,  because  reasons  may  exist  where  consolidation  might  not  be  desired,  it  may  be 
difficult  to  determine  if  such  a  condition  is  truly  unnecessary  without  interviewing  the  architect. 
Therefore,  a  subjective,  binary  assessment  is  made  by  the  evaluator  with  any  specific  items 
discovered  referred  to  the  program  for  their  consideration. 

3.4.6.  Evaluation  Measures  for  Readability:  OV  &  SV  Readability 

The  two  constructed,  proxy  measures  for  Readability  are  OV  READABILITY  and  SV 
READABILITY.  These  respectively  measure  whether  or  not  Operational  View  and  Systems  View 
information  are  presented  clearly  and  concisely.  They  are  subjective  evaluations  by  operational- 
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level  and  systems  engineer-level  subject  matter  experts.  Each  available  OV  and  SV  product 
should  be  reviewed  as  a  whole  and  subjectively  rated  readable/unreadable.  The  final  assessment 
is  a  percentage  of  readable  OV  or  SV  views  over  their  respective  total  available  OV  or  SV  views. 

3.4.7.  Evaluation  Measure  for  Scalability:  Scale 

The  constructed,  proxy  SCALE  measure  addresses  the  issue  of  whether  or  not  the  scale  of 
architecture  can  be  at  least  doubled  while  retaining  its  desired  function  and  properties  without 
significantly  increasing  complexity.  SCALE  is  a  subjective  assessment  of  all  available  products 
to  determine  if  none,  some,  most,  or  all  views  could  handle  double  the  nodes  without  undue 
complexity. 

3.4.8.  Evaluation  Measure  for  Tailorability:  Decomposition 

The  natural,  direct  DECOMPOSITION  measure  evaluates  the  degree  to  which  the 

architecture  can  be  tailored.  The  primary  source  for  this  measurement  is  the  Operational  Activity 

Model  (OV-5).  Many  levels  of  decomposition  are  indicative  of  a  high  level  of  Tailorability.  The 

possible  score  categories  are: 

o  None 
o  1  level 
o  2  levels 
o  3+  levels 

3.4.9.  Evaluation  Measure  for  Evolvability:  Tool  Format 

The  TOOL  FORMAT  measure  evaluates  the  degree  to  which  the  products  can  be  easily 
edited  to  handle  refinements  based  on  the  method  of  development.  It  is  a  constructed,  proxy 
measure  that  assesses  the  effect  of  one  input  in  relation  to  the  ability  to  reflect  the  input  through 
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all  views.  For  example,  Telelogic’s  System  Architect  architecture -building  software  can  carry  a 

single  input  throughout  multiple  views.  The  AV-1  should  be  reviewed  for  the  architecture 

development  tools  to  be  used.  If  not  specified,  the  file  format  of  the  available  views  should  be 

used.  The  possible  score  categories  are: 

o  In  general,  the  product  has  to  be  built  again  from  the  start 
o  In  general,  one  input  is  reflected  in  single  reference  (e.g.,  no  find  and  replace  in 
Microsoft  Powerpoint) 

o  In  general,  one  input  is  reflected  in  instant  view  references  but  not  other  views 
(e.g.,  Microsoft  Word's  find  and  replace  in  all  documents) 
o  In  general,  one  input  is  reflected  in  all  relevant  views  (e.g.,  a  System  Architect 
change  applies  to  multiple  views) 

3.4.10.  Evaluation  Measure  for  Compliancy:  DoDAF  Compliancy 

The  natural,  direct  DODAF  COMPLIANCY  measure  evaluates  the  percentage  of  architecture 
products  which  comply  with  DoDAF  standards.  Each  available  view  should  be  compared  to  the 
appropriate  DoDAF  description  to  assess  its  compliancy  (DoD,  2007b).  The  final  determination 
is  the  ratio  of  the  total  number  of  products  in  compliance  to  the  total  number  of  available 
products. 

3.4.11.  Evaluation  Measure  for  Traceability:  Requirements  Traceability 

The  natural,  direct  REQUIREMENTS  TRACEABILITY  measure  evaluates  the  degree  to  which 
requirements  are  met  by  functions/activities.  The  Operational  Activity  to  Systems  Function 
Traceability  Matrix  (SV-5a)  “depicts  the  mapping  of  operational  activities  to  system  functions 
and  thus  identifies  the  transformation  of  an  operational  need  into  a  purposeful  action  performed 
by  a  system”  (DoD,  2007b:  5-39).  Therefore,  the  creation  and  validation  of  an  SV-5  would 
accurately  measure  the  value  of  Traceability  by  the  percentage  of  operational  activities  mapped 
to  system  functions. 
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3.4.12.  Evaluation  Measures  for  Consistency:  Internal  &  External  Consistency 

The  INTERNAL  CONSISTENCY  measure  determines  if  each  available  product  is  in 
agreement  with  itself.  The  EXTERNAL  CONSISTENCY  measure  determines  if  each  available 
product  is  in  agreement  with  the  other  available  products.  Both  of  these  natural,  direct  measures 
are  determined  by  the  ratio  of  the  number  of  consistent  products  by  the  total  number  of  products 
available. 

3.4.13.  Evaluation  Measures  for  Subject  Matter  Expert  (SME)  Input 

3.4.13.1.  SME  Effectiveness 

The  constructed,  proxy  SME  EFFECTIVENESS  measure  evaluates  the  degree  of 
effectiveness  of  the  SME’s  involved  with  the  architecture  development.  This  is  determined  by 
examining  the  AV-1  for  any  plan  for  involving  SMEs  with  the  representation  of  effectiveness 
based  on  experience.  Specifically  for  this  effort,  a  SME  with  over  five  years  of  force  protection 
experience  was  specified  by  the  SEIWG  as  the  most  effective.  The  level  of  SME  experience 
may  be  easily  tailored  to  a  specific  program’s  need;  however,  the  same  five  year  specification 
may  be  left  as  the  default  for  the  general  case.  The  possible  score  categories  are: 
o  No  Plan 

o  Plan/No  SMEs  identified 
o  SMEs  identified  but  no  reference  to  experience 
o  Identified  SMEs  average  <  5  years  experience 
o  Identified  SMEs  average  >  5  years  experience 

3.4.13.2.  SME  Involvement 

The  natural,  direct  SME  INVOLVEMENT  measure  evaluates  the  number  of  SMEs  involved 
from  different  stakeholder  organizations.  Because  this  effort  is  a  joint  project,  the  SEIWG 
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specified  that  involvement  from  multiple  services  would  define  the  scoring  categories  (0,  1,  2,  3, 
4,  and  Multiple  SMEs  from  multiple  services).  By  default,  the  same  number  of  categories  may 
be  used  with  the  number  of  services  changed  to  number  of  organizations.  For  example,  the 
number  of  major  commands  involved  would  be  used  instead  of  services  for  an  Air  Force-level 
program.  The  possible  score  categories  are: 
o  No  involvement 

o  One  Stakeholder  Organization  SME 
o  Two  Stakeholder  Organization  SME 
o  Three  Stakeholder  Organization  SME 
o  Four  Stakeholder  Organization  SME 
o  Many  Stakeholder  SMEs  from  many  organizations 

3.5.  Create  Single  Dimension  Value  Functions 

These  measures  consisted  of  different  measurement  units  and  different  scales  (although 
most  here  are  categorical);  therefore,  Single  Dimension  Value  Functions  (SDVFs)  were  created 
to  convert  the  units  of  each  evaluation  measure  into  a  score  ranging  from  zero  to  unity.  This 
metric  allowed  for  easy  summation  into  an  overall  score.  These  value  functions  were  drafted  by 
the  authors  and  refined  and  validated  during  meetings  with  the  decision  maker  and  SMEs.  A 
summary  table  is  provided  in  Appendix  C  for  reference. 

The  worst  and  best  case  scenarios  for  each  measure  were  discussed  to  establish  the  values 
of  quality  boundary  (zero  and  one).  Key  intermediate  points  were  then  selected  for  each  measure 
with  values  assigned  by  the  decision  maker.  While  these  values  represent  the  joint  force 
protection  domain,  they  may  be  used  as  the  default  or  starting  point  to  tailor  according  to  the 
needs  of  another  program's  decision  maker. 

These  graphs  were  developed  using  Hierarchy  Builder  Version  1.01  (Weir,  2008).  This 

Microsoft  Excel  spreadsheet  plug-in  allows  quick  definition  of  the  value  functions  by  specifying 
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the  type  of  function  (e.g.,  monotonically  increasing  exponential)  and  the  pertinent  inflection 
points. 

3.5.1.  Access  Value  Function 

The  ACCESS  value  function  uses  a  discrete,  categorical  scale.  Figure  13  specifies  the 
value  the  decision  maker  placed  on  the  measure’s  categories  of  time  to  grant  access.  The 
decision  maker  specified  the  worst  case  scenario  (lower  bound,  assigned  a  value  of  zero)  to  be  no 
available  electronic  access  while  the  best  case  scenario  (upper  bound,  assigned  a  value  of  one)  is 
access  within  five  minutes.  The  other  categories  ranged  as  shown  in  Figure  13  according  to  the 
decision  maker’s  value. 
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Figure  13.  Access  Value  Function 
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3.5.2.  Product  Locatability  Value  Function 


The  PRODUCT  LOCATABILITY  value  function  uses  a  discrete,  categorical  scale.  Figure  14 
specifies  the  decision  maker’s  value  associated  with  how  quickly  the  desired  products  can  be 
located.  The  decision  maker  specified  the  worst  case  scenario  to  be  the  inability  to  locate  the 
products  while  the  best  case  scenario  is  locating  the  products  within  five  minutes.  The  other 
categories  ranged  as  shown  in  Figure  14  according  to  the  decision  maker’s  value. 
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Figure  14.  Product  Locatability  Value  Function 
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3.5.3.  Access  Control  Value  Function 


The  ACCESS  CONTROL  value  function  uses  a  discrete,  categorical  scale.  Figure  15 
specifies  the  decision  maker’s  value  associated  with  the  plan  and  implementation  of  the 
appropriate  level  of  access  protection  over  the  architecture  products.  The  decision  maker 
specified  the  worst  case  scenario  to  be  no  plan  or  an  inadequate  plan  while  the  best  case  scenario 
is  implementation  of  appropriate  protection.  The  other  categories  ranged  as  shown  in  Figure  15 
according  to  the  decision  maker’s  value. 
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Figure  15.  Access  Control  Value  Function 
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3.5.4.  Document  Protection  Value  Function 


The  DOCUMENT  PROTECTION  value  function  uses  a  discrete,  categorical  scale.  Figure  16 
specifies  the  decision  maker’s  value  associated  with  the  level  of  write-protection  measures  or 
configuration  control  in  place.  The  decision  maker  specified  the  worst  case  scenario  to  be  no 
write-protection  plan  or  configuration  control  plan  while  the  best  case  scenario  is  a  plan  exists 
and  all  products  controlled.  The  other  categories  ranged  as  shown  in  Figure  16  according  to  the 
decision  maker’s  value. 
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Figure  16.  Document  Protection  Value  Function 
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3.5.5.  File  Management  Value  Function 


The  FILE  MANAGEMENT  value  function  uses  a  discrete,  categorical  scale.  Figure  17 
specifies  the  decision  maker’s  value  associated  with  the  file  management  scenarios.  A  notable 
difference  in  this  value  function  is  that  the  categories  are  not  equally  incremental.  The  research 
team  initially  proposed  a  higher  0.25  value  for  a  file  management  system  that  was  complete  but 
not  maintained.  However,  the  decision  maker  determined  that  a  system  that  exists,  but  is  not 
complete  provides  the  same  value  (0.5)  as  one  that  exists,  but  is  not  regularly  maintained.  The 
decision  maker  specified  the  worst  case  scenario  to  be  no  file  management  system  while  the  best 
case  scenario  is  implementation  of  a  file  management  system  with  all  products  maintained.  The 
other  categories  ranged  as  shown  in  Figure  17  according  to  the  decision  maker’s  value. 
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Figure  17.  File  Management  Value  Function 
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3.5.6.  File  Format  Value  Function 


The  FILE  FORMAT  value  uses  a  discrete,  categorical  scale.  Figure  18  specifies  the  decision 
maker’s  value  associated  with  the  categories  regarding  the  file  formats  for  the  architecture 
products.  The  decision  maker  specified  the  worst  case  scenario  to  be  no  electronic  products  or 
inaccessible  products  while  the  best  case  scenario  is  products  in  a  general  file  format.  The  other 
categories  ranged  as  shown  in  Figure  18  according  to  the  decision  maker’s  value. 
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Figure  18.  File  Format  Value  Function 
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3.5.7.  Connections  Value  Function 


The  CONNECTIONS  value  function  uses  a  monotonically  increasing,  exponential  scale. 
Figure  19  specifies  the  decision  maker’s  value  associated  with  the  percent  of  products  with  easy 
to  understand  entities.  The  inflection  point  was  specified  as  0.3  on  the  value  axis  meaning  60 
percent  of  the  available  products  exist  as  such.  The  function  begins  to  earn  most  of  its  value  at 
the  >  0.6  (or  60  percent)  mark. 


Figure  19.  Connections  Value  Function 
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3.5.8.  Architecture  Redundancy  Value  Function 


The  ARCHITECTURE  REDUNDANCY  value  function  uses  a  discrete,  categorical  scale. 
Figure  20  specifies  the  decision  maker’s  value  associated  with  the  number  of  entities  found  to  be 
redundant.  The  decision  maker  specified  the  worst  case  scenario  to  be  greater  than  one 
redundancy  in  10  entities  while  the  best  case  scenario  is  less  than  one  redundancy  in  500  entities. 
The  other  categories  ranged  as  shown  in  Figure  20  according  to  the  decision  maker’s  value. 
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Figure  20.  Architecture  Redundancy  Value  Function 
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3.5.9.  Architecture  Economy  Value  Function 


The  ARCHITECTURE  ECONOMY  value  function  uses  a  discrete,  binary  scale.  Because  the 
measure  is  either  a  yes  or  a  no,  the  value  is  by  default  the  worst  and  best  values  of  zero  or  one, 
respectively.  This  function  is  shown  in  Figure  21. 


Figure  21.  Architecture  Economy  Value  Function 
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3.5.10.  OV  Readability  Value  Function 


The  OV  READABILITY  value  function  uses  a  monotonically  increasing,  S-curve  scale. 
Figure  22  specifies  the  decision  maker's  value  associated  with  the  percentage  of  readable  OVs. 
For  the  S-curve,  greater  value  is  earned  with  a  higher  percentage  of  readability  (inflection  point 
at  0.25)  on  the  bottom  end  of  the  curve  which  then  breaks  at  the  0.5  point  where  lesser  value  is 
earned  as  the  percentage  of  readability  increases  (inflection  points  specified  at  0.75). 


Figure  22.  OV  Readability  Value  Function 
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3.5.11.  SV  Readability  Value  Function 


The  SV  READABILITY  value  function  uses  a  monotonically  increasing  S-curve  exactly  the 
same  as  the  OV  READABILITY  SDVF  described  previously.  Figure  23  specifies  the  decision 
maker's  value  associated  the  percentage  of  readable  SVs.  For  the  S-curve,  greater  value  is 
earned  with  a  higher  percentage  of  readability  (inflection  point  at  0.25)  on  the  bottom  end  of  the 
curve  which  breaks  at  the  0.5  point  where  lesser  value  is  earned  as  the  percentage  of  readability 
increases  (inflection  points  specified  at  0.75). 


Figure  23.  SV  Readability  Value  Function 
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3.5.12.  Scale  Value  Function 


The  SCALE  value  function  uses  a  simple  a  discrete,  categorical  scale.  Figure  24  specifies 
the  decision  maker's  value  associated  with  the  ability  of  the  architecture  to  double  in  scale 
without  significantly  increasing  complexity.  The  decision  maker  specified  the  worst  case 
scenario  to  be  no  views  able  to  double  in  scale  while  the  best  case  scenario  is  all  views  can 
double  in  scale  without  significantly  increasing  complexity.  The  other  categories  ranged  as 
shown  in  Figure  24  according  to  the  decision  maker’s  value. 
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Figure  24.  Scale  Value  Function 
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3.5.13.  Decomposition  Value  Function 


The  DECOMPOSITION  value  function  uses  a  discrete,  categorical  scale.  Figure  25 
specifies  the  decision  maker's  value  associated  with  the  levels  of  decomposition  found  in  the 
OV-5.  The  decision  maker  specified  the  worst  case  scenario  to  be  no  decomposition  while  the 
best  case  scenario  is  decomposition  to  three  or  more  levels.  The  other  categories  ranged  as 
shown  in  Figure  25  according  to  the  decision  maker’s  value. 
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Figure  25.  Decomposition  Value  Function 
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3.5.14.  Tool  Format  Value  Function 


The  TOOL  FORMAT  value  function  uses  a  discrete,  categorical  scale.  Figure  26  specifies 
the  decision  maker's  value  associated  with  the  ability  of  the  tools  used  to  incorporate  changes. 
The  decision  maker  specified  the  worst  case  scenario  to  be  the  inability  of  a  tool  to  incorporate 
changes  thus  requiring  views  to  be  rebuilt  while  the  best  case  scenario  is  one  change  carried 
through  multiple  views.  The  other  categories  ranged  as  shown  in  Figure  26  according  to  the 
decision  maker’s  value. 
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Figure  26.  Tool  Format  Value  Function 
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3.5.15.  DoDAF  Compliancy  Value  Function 


The  DODAF  COMPLIANCY  value  function  uses  a  monotonically  increasing,  linear  scale. 
The  decision  maker's  value  of  the  percentage  of  products  that  comply  with  DoDAF  standards 
increases  linearly  as  the  percentage  of  products  in  compliance  increases.  This  is  shown  in  Figure 
27. 


Figure  27.  DoDAF  Compliancy  Value  Function 
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3.5.16.  Requirement  Traceability  Value  Function 


The  REQUIREMENT  TRACEABILITY  value  function  uses  a  monotonically  increasing, 
exponential  scale.  Figure  28  specifies  the  decision  maker's  value  corresponding  to  the  level  of 
completeness  of  the  SV-5.  For  this  exponential,  the  inflection  point  was  specified  at  the  point 
0.6,  representing  a  60  percent  complete  SV-5  and  resulting  in  a  value  of  0.2.  The  function  starts 
to  earn  value  more  quickly  at  the  >  0.6  mark. 


Figure  28.  Requirements  Traceability  Value  Function 
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3.5.17.  Internal  Consistency  Value  Function 


The  INTERNAL  CONSISTENCY  value  function  uses  a  monotonically  increasing,  S-curve 
scale.  Figure  29  specifies  the  decision  maker's  value  associated  with  the  percentage  of  products 
that  have  no  inconsistencies  within  themselves.  For  the  S-curve,  greater  value  is  earned  with  a 
higher  percentage  of  readability  (inflection  point  at  0.25)  on  the  bottom  end  of  the  curve  which 
breaks  at  the  0.5  point  where  lesser  value  is  earned  as  the  percentage  of  readability  increases 
(inflection  points  specified  at  0.75). 


Figure  29.  Internal  Consistency  Value  Function 
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3.5.18.  External  Consistency  Value  Function 


The  EXTERNAL  CONSISTENCY  value  function  uses  the  same  monotonically  increasing,  S- 
curve  as  the  previous  SDVF.  Figure  30  specifies  decision  maker's  value  associated  with  the 
percentage  of  products  with  no  inconsistencies  to  other  products.  For  the  S-curve,  greater  value 
is  earned  with  a  higher  percentage  of  readability  (inflection  point  at  0.25)  on  the  bottom  end  of 
the  curve  which  breaks  at  the  0.5  point  where  lesser  value  is  earned  as  the  percentage  of 
readability  increases  (inflection  points  specified  at  0.75). 


Figure  30.  External  Consistency  Value  Function 
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3.5.19.  SME  Effectiveness  Value  Function 


The  SME  EFFECTIVENESS  value  function  uses  a  discrete,  categorical  scale.  Figure  31 
specifies  the  decision  maker's  value  associated  with  whether  the  SMEs  have  been  identified  and 
how  much  experience  each  SME  has  to  contribute  to  the  project.  The  decision  maker  specified 
the  worst  case  scenario  to  be  no  plan  for  SMEs  while  the  best  case  scenario  is  identifying  SMEs 
with  an  average  of  over  five  years  experience.  The  other  categories  ranged  as  shown  in  Figure 
31  according  to  the  decision  maker’s  value. 
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Figure  31.  SME  Effectiveness  Value  Function 
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3.5.20.  SME  Involvement  Value  Function 


The  SME  INVOLVEMENT  value  function  uses  a  discrete,  categorical  scale.  Figure  32 
specifies  the  decision  maker's  value  associated  with  the  number  of  actual  SMEs  and  their 
organizations  involved.  The  decision  maker  specified  the  worst  case  scenario  to  be  no  SME 
involvement  while  the  best  case  scenario  is  involvement  by  multiple  SMEs  from  multiple 
organizations.  The  other  categories  ranged  as  shown  in  Figure  32  according  to  the  decision 
maker’s  value. 
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Figure  32.  SME  Involvement  Value  Function 
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3.6.  Weight  Architecture  Quality  Values  Hierarchy 


As  previously  discussed,  the  joint  force  protection  VDEA-Score  hierarchy  consisted  of 
multiple  categories  that  the  decision  maker  validated  as  valuable  to  architecture  quality.  These 
values  are  not  equally  essential,  however.  To  account  for  these  differences  in  importance,  a 
direct  weighting  technique  was  employed.  A  local  weight  described  how  much  weight  a  sub¬ 
value  contributed  to  the  value  above  it,  while  a  global  weight  described  how  much  weight  each 
of  the  last-tier  values  in  each  branch  of  the  value  hierarchy  contributed  to  the  overall  value  at  the 
top  of  the  hierarchy. 

The  first  tier  of  the  value  hierarchy  consists  of  the  two  overall  branches,  as  previously 
stated.  The  System  Effectiveness  Values  branch  focused  on  force  protection-specific  objectives, 
while  the  Architecture  Quality  Values  branch  focused  primarily  on  architecture-specific 
objectives.  The  decision  maker  placed  60  percent  (0.6  out  of  1.0)  importance  on  the  System 
Effectiveness  Values  and  40  percent  (0.4  out  of  1.0)  importance  on  the  Architecture  Quality 
Values  branch  as  shown  in  Figure  33.  These  weightings  of  importance  may  easily  be  tailored 
based  on  a  different  decision  maker's.  Again,  only  the  Architecture  Quality  Values  branch  was 
described  in  this  thesis.  The  weighted  System  Effectiveness  Values  branch  hierarchy  is  provided 
as  reference  (Mills,  2009)  in  Appendix  D.  The  Architecture  Quality  Values  hierarchy  and  their 
associated  local  and  global  weights  are  shown  in  Figure  34  where  “L”  is  for  local  and  “G”  is  for 
global  weights.  Table  8  also  provides  a  summary  listing  of  the  values  and  their  weights. 
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Figure  33.  VDEA-Score  Hierarchy  First  Tier  Showing  Local  Weights 


Figure  34.  Architecture  Quality  Values  Hierarchy  with  Weights 
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Table  8.  Architecture  Quality  Value  Weights 


Value 

Local 

Global 

Weight 

Weight 

Architecture  Quality  Values 

1.000 

0.400 

Accessibility 

0.250 

0.100 

Subscribability 

0.333 

0.033 

Controllability 

0.333 

0.033 

Protectability 

0.333 

0.033 

Usability 

0.350 

0.140 

Longevity 

0.300 

0.042 

Understandability 

0.700 

0.098 

Simplicity 

0.400 

0.039 

Readability 

0.600 

0.058 

Modil 

lability 

0.150 

0.600 

Scalability 

0.400 

0.024 

Tailorability 

0.400 

0.024 

Evolvability 

0.200 

0.012 

Accountability 

0.250 

0.100 

Compliancy 

0.300 

0.030 

Traceability 

0.200 

0.020 

Consistency 

0.200 

0.020 

SME  Input 

0.300 

0.030 

3.6.1.  Local  Weights  for  Second-Tier  Values 

The  values  comprising  the  second  tier  of  the  hierarchy  under  the  Architecture  Quality 
Values  branch  were  the  four  values  determined  most-essential  in  regards  to  the  quality  of 
architecture.  Thirty-five  percent  local  importance  (0.35  out  of  1.0)  was  placed  on  Usability. 
Twenty- five  percent  importance  (0.25  out  of  1 .0)  was  placed  on  both  Accessibility  and 
Accountability.  The  remaining  fifteen  percent  (0.15  out  of  1.0)  was  placed  on  Modifiability.  The 
weights  assigned  to  the  values  comprising  the  third-tier  values  are  discussed  in  the  following 
sections. 
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3.6.1. 1.  Local  Weights  for  Accessibility  Sub-Values 


Calculating  how  much  weight  the  third-tier  values  Subscribability,  Controllability,  and 
Protectability  contribute  to  the  second-tier  objective  Accessibility  was  a  fairly  simple  process. 
All  three  values  were  assessed  as  equally  important  to  Accessibility,  thus  they  were  all  equally 
weighted  at  0.333  out  of  1.0.  This  distribution  is  displayed  in  Figure  35. 


Figure  35.  Local  Weights  for  Accessibility  Sub-Values 
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3.6.I.2.  Local  Weights  for  Usability  Sub-Values 


The  decision  maker  concluded  that  for  Usability ,  Understandability  was  more  than  twice 
as  important  as  Longevity,  and  a  70  percent  importance  (0.7  out  of  1.0)  was  placed  on  it.  The 
remaining  30  percent  (0.3  out  of  1 .0)  went  to  Longevity  as  shown  in  Figure  36.  Next, 
Readability  was  assessed  as  more  important  than  Simplicity ,  which  received  60  percent  (0.6  out 
of  1.0)  emphasis  on  it.  The  remaining  40  percent  (0.4  out  of  1.0)  was  placed  on  Simplicity. 


Figure  36.  Local  Weights  for  Usability  Sub-Values 
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3.6.I.3.  Local  Weights  for  Modifiability  Sub-Values 


To  determine  how  much  weight  Scalability,  Evolvability,  and  Tailor  ability  contribute  to 
Modifiability,  the  decision  maker  first  indicated  that  Evolvability  was  least  valued  because  of  the 
unlikely  chance  the  products  would  be  developed  in  a  non-standard  format.  They  determined 
that  Scalability  and  Tailorability  were  equal  in  importance  to  Modifiability,  but  also  that  they 
were  twice  as  important  as  Evolvability.  This  corresponds  to  a  40  percent  importance  (0.4  out  of 
1.0)  granted  to  both  Scalability  and  Tailorability,  and  the  remaining  20  percent  (0.2  out  of  1.0) 
placed  on  Evolvability  as  shown  in  Figure  37. 


Figure  37.  Local  weights  for  Modifiability  Sub-Values 
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3.6.I.4.  Local  Weights  for  Accountability  Sub-Values 


To  determine  how  much  weight  Compliancy,  Traceability,  Consistency,  and  SME  Input 
contribute  to  Accountability,  the  decision  maker  first  indicated  that  Traceability  and  Consistency 
were  less  valued,  though  equally  important,  than  Compliancy  and  SME  Input.  The  decision 
maker  also  stated  that  Compliancy  and  SME  Input  were  equally  important  and  that  they  were  1.5 
times  more  important  than  Traceability  and  Consistency.  This  corresponds  to  a  30  percent 
importance  (0.3  out  of  1.0)  placed  on  both  Compliancy  and  SME  Input,  and  the  remaining  40 
percent  split  evenly  (0.2  out  of  1.0  each)  between  Traceability  and  Consistency  as  shown  in 
Figure  38. 


Figure  38.  Local  Weights  for  Accountability  Sub-Values 
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3.6.I.5.  Local  Weights  for  Measurements 


It  should  also  be  noted  that  six  of  the  sub-values  possessed  multiple  measures.  All  were 
of  equal  weight  except  the  ACCESS  measure  which  was  valued  twice  as  much  as  PRODUCT 
LOCAT ABILITY  because  a  user  could  not  locate  the  products  if  access  was  unavailable. 
Therefore,  the  ACCESS  measure  had  a  0.67  weight  compared  to  PRODUCT  LOCAT  ABILITY 's  0.33. 

3.6.2.  Verification  of  Weights 

To  help  the  decision  maker  validate  that  proper  weights  were  assigned  to  the  values, 
tornado  graphs  were  used  to  provide  better  visualization  of  the  value  rankings  by  the  applied 
weights.  The  decision  maker  reviewed  these  decisions  and  validated  that  the  values  fell  in  the 
proper  place  in  comparison  at  the  global  level.  Initially,  only  the  local  weights  were  discussed. 
The  graphs  were  then  used  to  show  the  global  weights  so  the  decision  maker  could  visually  rank 
the  importance  of  each  value.  A  top-down  approach  was  used,  from  the  first-tier  values  and 
descending  down  the  hierarchy.  The  top  graph  showing  the  global  weights  for  the  first-tier  is 
shown  in  Figure  39.  Note  that  the  two  weights  sum  to  1.0  (100  percent). 
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Figure  39.  Tier  1  Global  Weights 
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Accessibility,  Usability,  Modifiability,  and  Accountability  were  assigned  the  weights  of 
0.25,  0.35,  0.15,  and  0.25  respectively.  The  graph  displaying  the  global  weights  of  the  second 
tier  of  the  Architecture  Quality  Values  branch  is  shown  in  Figure  40.  Again,  note  that  the  global 
weights  sum  to  0.4  (the  total  Architecture  Quality  Value  weight). 
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Figure  40.  Tier  2  Architecture  Quality  Value  Global  Weights 


The  Tier  2  values  were  decomposed  into  their  Tier  3  values  for  further  verification  as 
displayed  in  Figure  41.  As  shown,  the  attribute  most  valuable  overall  to  an  architecture  is 
Understandability  as  the  graph  displays  its  rank  as  over  two  times  as  important  as  the  next  most 
valuable  attribute  Longevity.  These  graphs  allowed  the  decision  maker  to  adjust  local  weights  to 
accurately  reflect  the  importance  rankings  of  the  value  categories. 
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Tier  3  Architecture  Quality  Global  Weight 


Figure  41.  Tier  3  Architecture  Quality  Value  Global  Weights 


3.7.  Model  Preliminary  Validation  Efforts 

Additionally,  because  this  thesis’s  focus  within  the  overall  portion  of  the  joint  force 
protection  project  was  to  produce  a  VDEA-Score  model  for  evaluating  architectures,  the  authors 
assessed  the  model's  effectiveness  by  using  another  program’s  architecture.  The  authors 
reviewed  the  Air  Force  Architecture  Repository  for  potential  candidate  architectures  and  selected 
the  Information  and  Resource  Support  System  (IRSS).  This  was  a  database  environment  for 
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collaborative  requirements  and  planning  providing  Air  Force  agencies  access  to  planning, 
requirements,  and  financial  data  (Zechar,  2006).  This  particular  architecture  was  chosen  for  its 
maturity  and  the  relatively  large  number  of  available  products  (AV-1,  AV-2,  OV-1,  OV-2,  OV-3, 
OV-5,  OV-6c,  OV-7,  SV-1,  SV-5,  SV-6,  SV-7,  SV-8,  and  TV-1).  The  results  of  this  validation 
are  presented  in  the  next  chapter. 

3.8.  Alternative  Generation 

One  of  the  purposes  of  VFT  is  to  facilitate  comparison  of  alternatives  to  make  better 
informed  decisions.  Because  this  joint  force  protection  effort  is  a  work-in-progress  and  only  the 
initial  architecture  exists,  no  alternatives  were  available.  Likewise,  there  was  no  need  to 
generate  actual  alternatives  at  this  point  as  the  effort  is  focused  on  evaluating  the  current  draft 
architecture  to  identify  areas  to  improve  before  finalizing  the  products  for  Milestone  B. 

Therefore,  theoretical  alternatives  were  generated  based  on  the  areas  of  improvement  that  were 
identified.  This  identification  process  results  from  the  model  evaluation  and  a  subsequent 
analysis  on  the  measures.  This  measurement  analysis  examines  the  impact  of  varying  a  single 
measure's  score  on  the  overall  score  while  keeping  the  other  measures'  scores  as  evaluated.  This 
analysis  identifies  the  areas  of  strength  and  weakness  by  observing  the  greatest  decrease  or 
increase  respectively  by  varying  each  measure's  score.  The  measures  showing  the  greatest 
potential  increase  in  score  are  considered  prime  candidates  for  developing  alternatives  based  on 
improving  the  architecture  in  the  affected  area.  These  alternatives  are  representative  value 
scored  architectures  to  demonstrate  the  higher  scores  attainable  by  improving  in  the  noted  areas. 
The  results  are  presented  in  the  next  chapter. 
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3.9.  Summary 


The  development  of  the  complete  Architecture  Quality  Values  hierarchy  within  the 
VDEA-Score  model  was  explained  in  this  chapter.  Additionally,  brief  descriptions  of  the 
additional  model  verification  effort  and  the  joint  force  protection  alternative  architecture 
generation  process  were  provided.  Analysis  of  the  architectures,  along  with  conclusions  and 
recommendations,  follow  in  the  remaining  chapters. 
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IV.  Results  and  Analysis 


With  the  value  hierarchy  defined,  associated  measures  determined,  value  functions 
assigned,  and  appropriate  weighting  factors  applied,  the  joint  force  protection  Value  Driven 
Enterprise  Architecture  Score  (VDEA-Score)  model  was  now  complete.  The  deterministic  and 
sensitivity  analysis  performed  on  the  Architecture  Quality  Value  hierarchy  is  presented  in  this 
chapter.  The  final  VDEA-Score  result  for  the  System  Effectiveness  Value  branch  from  Osgood 
(2009)  is  also  provided  to  show  the  complete  joint  force  protection  VDEA-Score. 

The  primary  analysis  was  completed  using  architecture  views  provided  on  24  December 
2008  by  the  642d  Electronic  Systems  Squadron  (ELSS)  for  AFIT’s  evaluation:  AV-1,  OV-1, 
OV-2,  OV-4,  OV-5,  OV-6c,  SV-1,  SV-2,  SV-4,  SV-6,  SV-lOc,  and  TV-1.  With  the  exception  of 
the  OV  READABILITY  measure,  the  authors  examined  these  twelve  products  as  the  evaluator 
applying  the  model  to  determine  the  VDEA-Score.  Measurement  analysis  was  conducted  which 
lead  to  development  of  the  theoretical  alternative  architectures  as  a  comparison  of  score 
improvement  for  addressing  deficient  areas.  Weight  sensitivity  analysis  was  also  conducted 
varying  the  value  weights.  Finally,  the  VDEA-Score  process  was  further  verified  by  assessing 
the  Information  and  Resource  Support  System  (IRSS)  architecture  to  determine  its  Tier  I 
Architecture  Quality  Value  VDEA-Score. 

4.1.  Joint  Force  Protection  VDEA-Scoring 

This  section  documents  the  initial  results  of  the  Architecture  Quality  Value  model  and 

provides  feedback  to  the  decision  maker  regarding  the  overall  quality  of  their  architecture. 

Specifically,  this  evaluation  highlights  the  values  and  measures  which  earned  the  most  value  in 
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the  overall  Architecture  Quality  Value  VDEA-Score  as  well  as  the  areas  for  improvement.  The 
analysis  also  addresses  the  impact  on  the  final  rankings  by  measures  having  relatively  high 
global  weights.  These  results  are  summarized  in  Table  9  at  the  end  of  this  section. 

4.1.1.  Access 

The  primary  source  for  evaluating  the  ACCESS  measure  for  the  value  of  Subscribability  is 
the  AV-1.  The  evaluator  found  no  information  related  to  this  measure  within  the  AV-1,  so  the 
alternative  proxy  evaluation  of  the  program’s  repository  was  used.  Based  on  the  evaluator's 
experience  requesting  access  to  the  program's  repository  web  site,  ACCESS  was  categorically 
evaluated  as  "3dy<access<lwk"  resulting  in  a  corresponding  value  score  from  the  SDVF  of 
0.500. 

4.1.2.  Product  Locatability 

The  primary  source  for  evaluating  the  PRODUCT  LOCATABILITY  measure  for  the  value  of 
Subscribability  is  the  AV-1.  The  evaluator  found  no  information  related  to  this  measure  within 
the  AV-1,  so  the  alternative  proxy  evaluation  of  the  program’s  repository  was  used.  Based  on 
the  evaluator's  experience  navigating  the  program's  repository  web  site,  PRODUCT 
LOCATABILITY  was  categorically  evaluated  as  "<5min"  resulting  in  a  corresponding  value  score 
from  the  SDVF  of  one. 

4.1.3.  Access  Control 

The  primary  source  for  evaluating  the  ACCESS  CONTROL  measure  for  the  value  of 
Protectability  is  the  AV-1.  The  evaluator  found  no  information  related  to  this  measure  within  the 
AV-1.  Therefore,  the  alternative  proxy  evaluation  of  the  program’s  repository  was  used.  Based 
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on  the  evaluator's  experience  following  the  instructions  to  establish  a  user  identification  and 
password  for  the  repository  web  site,  ACCESS  CONTROL  was  categorically  evaluated  as 
"Appropriate  Control"  resulting  in  a  corresponding  value  score  from  the  SDYF  of  one. 

4.1.4.  Document  Protection 

The  primary  source  for  evaluating  the  DOCUMENT  PROTECTION  measure  for  the  value  of 
Controllability  is  the  AV-1.  The  evaluator  found  no  information  related  to  this  measure  within 
the  AV-1.  Therefore,  the  alternative  proxy  evaluation  of  the  program’s  repository  was  used. 

The  evaluator  accessed  various  documents  and  attempted  to  change  them  on  the  repository  web 
site.  This  was  unsuccessful  as  appropriate  write  protections  were  in  place.  Based  on  this 
experience,  DOCUMENT  PROTECTION  was  categorically  evaluated  as  "Products  Controlled" 
resulting  in  a  corresponding  value  score  from  the  SDVF  of  one. 

4.1.5.  File  Management 

The  primary  source  for  evaluating  the  FILE  MANAGEMENT  measure  for  the  value  of 
Longevity  is  the  AV-1.  The  evaluator  found  no  information  related  to  this  measure  within  the 
AV-1.  Therefore,  the  alternative  proxy  evaluation  of  the  program’s  repository  was  used.  Based 
on  the  evaluator's  experience  examining  the  file  structure  on  the  repository  web  site,  the  folders 
demonstrated  organization  but  did  not  appear  to  demonstrate  obvious  implementation  of  an 
official  file  management  system.  Therefore,  FILE  MANAGEMENT  was  categorically  evaluated  as 
"System  does  not  exist"  resulting  in  a  corresponding  value  score  from  the  SDYF  of  zero. 
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4.1.6.  File  Format 


For  the  FILE  FORMAT  measure,  the  AV-1  specified  the  tools  to  be  used  for  development 
as  Telelogic's  System  Architect  and  Microsoft  Office.  Additionally,  the  available  products  were 
produced  by  these  tools.  The  evaluator  considered  these  tools  as  accepted  standards  capable  of 
producing  “General  File  Formats”  resulting  in  a  value  score  of  one. 

4.1.7.  Connections 

For  the  CONNECTIONS  measure,  the  evaluator  reviewed  each  product  and  assessed  the 
extent  to  which  the  connections  which  were  sufficiently  organized,  easy  to  follow,  and  made 
sense  to  the  reader.  Two  products  stood  out  as  not  meeting  these  criteria.  First,  the  SV-1  was 
noted  to  have  a  few  merged  needlines  which  were  difficult  to  trace  even  when  zoomed  in  to  a 
high  degree.  Second,  the  SV-4  also  had  numerous  needlines  merging  together  as  well  as 
numerous  unlabeled  needlines  making  them  difficult  to  trace.  Therefore,  CONNECTIONS  was 
determined  to  meet  10  out  of  the  12  total  products  resulting  in  a  corresponding  value  score  from 
the  SDVF  of  0.620. 

4.1.8.  Architecture  Redundancy 

For  the  ARCHITECTURE  REDUNDANCY  measure,  the  evaluator  did  not  note  any  entity, 
activities,  links,  etc.,  which  appeared  to  unnecessarily  accomplish  the  same  goal.  Therefore, 
ARCHITECTURE  REDUNDANCY  was  evaluated  categorically  as  "<1:10"  resulting  in  a 
corresponding  value  score  of  one. 
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4.1.9.  Architecture  Economy 


For  the  ARCHITECTURE  ECONOMY  measure,  the  evaluator  noted  no  obvious  instances  of 
multiple  activities  or  entities  being  used  when  they  could  be  consolidated.  This  was  a 
significantly  subjective  assessment  because  the  evaluator  lacked  sufficient  force  protection 
experience  to  identify  potential  system-related  instances.  In  terms  of  architecture  description 
instances,  the  choice  to  show  expanded  detail  for  example  within  a  system  block  on  the  SV-4 
was  considered  by  the  evaluator  to  be  appropriate  within  the  architect's  discretion.  Therefore, 
ARCHITECTURE  ECONOMY  was  evaluated  by  the  binary  category  "No  instances  found"  resulting 
in  the  value  score  of  one. 

4.1.10.  OV  Readability 

For  the  OV  READABILITY  measure,  a  career  security  forces  Subject  Matter  Expert  (SME) 
from  the  Security  Equipment  Integration  Working  Group  (SEIWG)  provided  additional  insight 
for  the  evaluation  from  a  security  forces  operational  perspective.  Each  of  the  five  OV  products 
was  examined.  With  the  exception  of  the  OV-5,  the  remaining  products  were  determined  overall 
to  be  easily  read.  The  OV-5,  by  virtue  of  its  extreme  detail  and  large  number  of  entities,  required 
a  significant  amount  of  zooming  in  and  alternating  views  to  read.  Therefore,  OV  READABILITY 
was  determined  to  have  "4  out  of  5"  readable  products  resulting  in  a  corresponding  SDVF  value 
score  of  0.930. 

4.1.11.  SV  Readability 

Like  the  previous  measure,  the  SV  READABILITY  measure  was  applied  to  the  five 

provided  SV  products.  Two  concerns  were  noted  to  reading  these  products.  First,  the  SV-2  had 

several  needlines  with  overlapping  text.  Secondly,  it  was  noted  the  SV-4  required  the  use  of  a 
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plotter  to  print  out  at  a  readable  size  or  zooming  and  panning  to  read  on  the  computer  screen. 
However,  the  readability  issue  with  the  SV-4  was  the  overlapping  text  in  the  "Indigo  Vision 
Control  Center  Software  Client"  block.  Therefore,  SV  READABILITY  was  determined  to  have  "3 
out  of  5"  readable  products  resulting  in  a  corresponding  SDVF  value  score  of  0.730. 

4.1.12.  Scale 

The  SCALE  measure  was  applied  to  all  available  products  to  determine  if  doubling  the 
number  of  nodes  would  greatly  increase  the  complexity.  This  measure  is  a  fairly  subjective 
assessment  by  the  evaluators  who  determined  categorically  "Most"  of  the  products  were  scalable 
resulting  in  the  value  score  of  0.600. 

4.1.13.  Decomposition 

The  DECOMPOSITION  measure  was  evaluated  by  reviewing  the  number  of  decomposition 
levels  in  the  OV-5.  Because  this  product  had  seven  levels  of  decomposition,  the  measure  was 
determined  to  be  categorically  "3+"  resulting  in  the  value  score  of  one. 

4.1.14.  Tool  Format 

The  TOOL  FORMAT  measure  was  applied  by  reviewing  the  AV-1  for  the  tools  used  to 
create  each  provided  view.  Because  Telelogic's  System  Architect  was  specified,  the  evaluator 
considered  this  a  common  tool  which  enforces  DoDAF  view  consistency  and  allows  easy  editing 
by  carrying  changes  through  multiple  views.  Therefore,  TOOL  FORMAT  was  determined 
categorically  to  be  "Input  carries  through  multiple  views"  resulting  in  a  value  score  of  one. 
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4.1.15.  DoDAF  Compliancy 


The  DODAF  COMPLIANCY  measure  was  applied  by  examining  each  available  view 
according  to  the  DoDAF  Vol  II,  version  1.5  (2008).  The  evaluator  noted  two  exceptions.  First, 
the  OV-2  needlines  show  how  information  is  exchanged  (e.g.,  LAN,  GIG),  whereas  the  DoDAF 
specifically  states  these  should  show  what  information  is  exchanged  (e.g.,  situational  awareness). 
Secondly,  the  SV-6  contains  a  good  amount  of  detail,  but  lacks  a  significant  amount  of  the 
descriptive  information  called  for  in  the  DoDAF  (e.g.,  no  information  on  Information  Assurance, 
Security,  Nature  of  Transaction,  or  Performance).  Therefore,  the  DODAF  COMPLIANCY  measure 
received  "10  out  of  12"  products  in  compliance  resulting  in  a  corresponding  SDVF  value  of 
0.830. 

4.1.16.  Requirements  Traceability 

The  REQUIREMENTS  TRACEABILITY  measure  required  reviewing  the  SV-5.  However,  an 
SV-5  was  not  provided,  resulting  in  0  percent  corresponding  to  a  value  score  of  zero. 

4.1.17.  Internal  Consistency 

The  INTERNAL  CONSISTENCY  measure  required  that  each  product  be  examined  for  any 
data  inconsistencies  within  itself.  Examining  each  entity,  function,  and  needline,  the  evaluator 
noted  two  product  exceptions:  SV-1  and  SV-4.  First,  the  SV-1  had  a  needline  label  ("54")  which 
was  far  removed  from  the  actual  associated  needline.  Second,  the  SV-4  had  several 
discrepancies: 

•  From  IA-4  Figure  3  (DfD,  Discoverii  Data  Conversion),  "Discoverii  Video 
Motion  JPEG  (for  AXIS  only)"  needline  not  on  the  master  view 

•  From  IA-4  Figure  7  (DfD,  UGS  Data  Conversion),  "Fetch  TRSS  generated 
Image"  needline  not  on  the  master  view 
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•  From  IA-4  Figure  7  (DfD,  UGS  Data  Conversion),  it  was  not  clear  that  TRSS 
RICC  and  TRSS  HHM  were  only  part  of  the  decomposed  entity  (e.g.,  should  be  a 
different  color  for  consistency  with  other  decomposed  entities) 

•  From  IA-4  Figure  9  (DfD,  TASS  Data  Conversion),  "TASS  Power"  needline  not 
on  the  master  view 

•  Numerous  needline  termination  arrows  depicted  in  different  colors  or  styles  (e.g., 
"PIR  Detection"  from  PIR  Sense  Transmit  to  Vindicator  shows  an  external  input 
arrow  head) 

•  Needline  label  "BFT  Location  ID  reports  RF"  significantly  distanced  from 
associated  needline 

Therefore,  INTERNAL  CONSISTENCY  was  evaluated  as  "10  out  of  12"  products  in  compliance 
resulting  in  a  corresponding  SDVF  value  of  0.950. 

4.1.18.  External  Consistency 

For  the  EXTERNAL  CONSISTENCY  measure,  each  individual  product  was  compared  for 
data  inconsistencies  to  every  other  product.  The  evaluator  noted  two  product  exceptions:  OV-2 
and  SV-6.  First,  not  all  of  the  OV-2  operational  nodes  were  depicted  as  system  nodes  in  the  SV- 
1.  Specifically,  the  Combat  Support  Node  was  conspicuously  absent.  Secondly,  the  SV-6  was 
missing  the  "PIR  Detection"  needline  described  on  the  SV-4's  PIR  Sense  Transmit  to  Vindicator 
blocks.  Therefore,  EXTERNAL  CONSISTENCY  was  evaluated  to  have  "10  out  of  12"  products  in 
compliance  resulting  in  a  corresponding  SDVF  value  of  0.950. 

4.1.19.  SME  Effectiveness 

For  the  SME  EFFECTIVENESS  measure,  the  AV-1  was  reviewed  for  any  information 
describing  a  plan  for  SME  involvement  with  any  requirement  for  experience.  No  information 
was  found.  Therefore,  SME  EFFECTIVENESS  was  evaluated  as  "No  information  provided" 
resulting  in  a  value  score  of  zero. 
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4.1.20.  SME  Involvement 


For  the  SME  INVOLVEMENT  measure,  the  AV-1  was  likewise  reviewed  for  any 
information  for  the  number  of  SMEs  involved  and  specifically  any  different  stakeholder 
organizations  represented  by  them.  While  the  AV-1  did  note  several  stakeholder  organizations 
in  paragraph  2.c.,  it  was  only  a  list  with  no  additional  detail  in  terms  of  roles,  responsibilities,  or 
involvement.  Therefore,  SME  INVOLVEMENT  was  evaluated  as  "No  information  provided" 
resulting  in  a  value  score  of  zero. 

4.1.21.  Joint  Force  Protection  Architecture  Quality  VDEA-Score  Summary 

The  final  step  in  providing  a  single  Architecture  Quality  Value  VDEA-Score  requires  the 
summation  of  each  of  the  individual  value  scores  according  to  their  respective  global  weights 
using  the  general  additive  value  function  of  v(x)  =  Y1f=1  (Xj).  Table  9  is  provided  as  a 

summary  of  these  individual  scores  with  the  resulting  vAq  of  0.287.  Thus,  the  Tier  I, 
Architecture  Quality  Value  branch  earned  0.287  points  out  of  the  total  possible  0.400  joint  force 
protection  VDEA-Score  points.  This  score  translates  to  a  local  or  normalized  (|| vAQ  ||  = 
vAq /0.400  )  0.718  (or  71.8  percent)  for  its  potential  value  in  this  portion  of  the  model.  Table  10 
shows  the  detail  of  the  value  category  scores  by  local  value  earned  and  percent  of  potential  local 
value  earned. 
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Table  9.  Joint  Force  Protection  Architecture  Quality  VDEA-Scoring 


Measure 

Assessment 

Global 

Weight 

At 

Value 

Score 

Vi(xL) 

Product 

A-iVi  (X() 

Access 

Proxy  Eval  of  Repository: 

3  day  <  access  <  1  week 

0.022 

0.500 

0.011 

Product  Locatability 

Proxy  Eval  of  Repository: 

<  5  minutes 

0.011 

1.000 

0.011 

Access  Control 

Proxy  Eval  of  Repository: 
Appropriate  Control 

0.033 

1.000 

0.033 

Document  Protection 

Proxy  Eval  of  Repository: 
Products  Controlled 

0.033 

1.000 

0.033 

File  Management 

Proxy  Eval  of  Repository: 
System  does  not  exist 

0.021 

0.000 

0.000 

File  Format 

General  File  Format 

0.021 

Connections 

10  out  of  12 

0.013 

Architecture  Redundancy 

0  redundancy  instances  found 

0.013 

Architecture  Economy 

No  instances  of  possible 
consolidation  found 

0.013 

1.000 

0.013 

OV  Readability 

4  out  of  5 

EESa 

SV  Readability 

3  out  of  5 

WBEM 

Scale 

Most  scalable  2X 

0.024 

0.600 

0.0144 

Decomposition 

3+  levels 

0.024 

Tool  Format 

Input  carries  thru  multiple  views 

0.012 

DoDAF  Compliancy 

10  out  of  12 

0.030 

0.0249 

Requirement  Traceability 

0%  (no  SV-5  provided) 

0.022 

0.000 

Internal  Consistency 

10  out  of  12 

0.010 

0.0095 

External  Consistency 

10  out  of  12 

0.010 

0.0095 

SME  Effectiveness 

No  info  provided 

0.015 

0.000 

SME  Involvement 

No  info  provided 

0.015 

0.000 

0.000 

n 

vAq(x)  =  'YJAivi(xi)  = 
i= 1 

0.287 
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Table  10.  Architecture  Quality  Value  VDEA-Score  Value  Earned 


Value  (h-iocai) 

Local  Value 
Earned 

%  of  Potential 
Local  Value 

Architecture  Quality  Values  (.4) 

0.287 

71.8% 

Accessibility  (.25) 

0.222 

88.8% 

Subscribability  (.333) 

0.222 

66.7% 

Protectability  (.333) 

0.333 

100.0% 

Controllability  (.333) 

0.333 

100.0% 

Usability  (.35) 

0.260 

74.2% 

Longevity  (.3) 

0.150 

50.0% 

Understandability  (.7) 

0.592 

84.6% 

Simplicity  (.5) 

0.349 

69.8% 

Readability  (.5) 

0.497 

99.4% 

Modifiability  (.15) 

0.126 

84.0% 

Scalability  (.4) 

0.240 

60.0% 

Tailorability  (.4) 

0.400 

100.0% 

Evolvability  (.2) 

0.200 

100.0% 

Accountability  (.25) 

0.110 

44.0% 

Compliancy  (.3) 

0.250 

83.3% 

Traceability  (.2) 

0.000 

0.0% 

Consistency  (.2) 

0.190 

95.0% 

SME  Input  (.3) 

0.000 

0.0% 

4.1.22.  Architecture  Quality  Value  Score  Analysis 

Figure  42  shows  graphically  the  VDEA-Score  for  Architecture  Quality  Value.  This 
graph  compares  the  value  earned  by  the  joint  force  protection  architecture  for  Architecture 
Quality  Value  over  the  full  potential  Architecture  Quality  Value.  Each  colored  block  represents 
the  value  earned  by  each  measure.  The  gaps  (white  spaces)  highlight  the  measures  earning  less 
than  full  value  as  areas  for  improvement.  The  blocks  are  presented  in  order  from  left  to  right 
starting  with  the  top  row  of  the  legend. 

This  comparison  graph  (Figure  42)  reiterates  the  previously  discussed  evaluation  results 
where  the  largest  value  gaps  reside  in  the  Tier  II  Accountability  branch  because  no  value  was 
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earned  for  REQUIREMENT  TRACEABILITY,  SME  EFFECTIVENESS,  and  SME  INVOLVEMENT.  These 
three  measures  accounted  for  the  lost  0.560  (56  percent)  of  the  total  potential  local  value  for  this 
Tier  II  branch.  Figure  43  shows  the  Accountability  branch  earned  0.439  (almost  44  percent)  of 
the  total  local  potential  value. 

Overall,  the  majority  of  value  was  lost  in  the  Accountability  branch.  The  scores  for  the 
other  branches  were  higher  with  Accessibility ,  Usability,  and  Modifiability  branches  earning 
0.888,  0.742,  and  0.840  of  their  potential  Tier  II  branch  value,  respectively.  Figures  44-46 
graphically  show  the  local  value  earned  by  each  of  these  branches. 


0  0.05 

"Access 
"Connections 
"  External  Consistency 
"OV  Readability 
SME  Involvement 


0.1  0.15 

"Access  Control 
"  Decomposition 
"  File  Format 
Product  Locatability 
"  SV  Readability 


0.2  0.25 

"Architecture  Economy 
"  Document  Protection 
File  Management 
Scale 

Tool  Format 


0.3  0.35  0.4 

"Architecture  Redundancy 
"  DoDAF  Compliancy 
"  Internal  Consistency 
SME  Effectiveness 
Requirement  Traceability 


Figure  42.  Joint  Force  Protection  VDEA-Score  vs  Potential  VDEA-Score 
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Accountability  0.439 


0.249 


0.0 


0.0 


0.0 


0.095 


0.095 


□  DoDAF  Compliancy 

□  SME  Effectiveness 

□  Internal  Consistency 


n  Requirement  Traceability 
1=1  SME  Involvement 
External  Consistency 


Figure  43.  Accountability  Local  Measure  Scores 


Accessibility  0.888 


0.222 


0.333 


0.333 


□  Subscribability  ■  Protectability  □  Controllability 


Figure  44.  Accessibility  Local  Sub-Tier  Value  Scores 
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Usability  0.742 


0.592 


0.150 


□  Understandability  D  Longevity 


Figure  45.  Usability  Local  Sub-Tier  Value  Scores 


Modifiability  0.840 

0.24 

0.4 

0.2 

□  Scalability  □  Tailorability  □  Evolvability 
Figure  46.  Modifiability  Local  Sub-Tier  Value  Scores 

From  this  evaluation,  several  areas  were  identified  to  assist  the  program  office  with  areas 
of  improvement  that  could  raise  the  overall  Architecture  Quality  Values  score.  Most  of  the  items 
noted  in  the  previous  analysis  section  only  require  minor  changes  to  potentially  increase  the 
VDEA-Score  to  its  full  potential  for  this  Tier  I  branch.  The  areas  with  the  most  work  required  in 
order  of  the  authors’  estimate  of  effort  involved  are: 
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1 .  Traceability  -  Requires  development  of  an  SV-5 

2.  Compliancy  -  Requires  a  number  of  additional  data  fields  in  the  SV-6 

3.  Longevity  -  Requires  development  of  a  file  management  plan  and  documentation 
in  the  AV-1 

While  given  full  value  in  the  evaluation,  the  areas  of  Subs  crib  ability,  Protectability,  and 
Controllability  would  also  benefit  from  reference  in  the  AV-1  to  allow  more  direct  evaluation. 
These  were  scored  full  value  by  proxy  evaluation  of  the  program's  on-line  repository.  However, 
the  AV-1  is  a  very  flexible  document  allowing  a  multitude  of  useful  information  concerning  the 
program  and  specifically  the  architecture.  More  detail  regarding  the  SMEs  in  the  AV-1  would 
increase  the  value  in  the  overall  score  because  these  were  scored  zero. 

It  is  also  interesting  to  note  that  in  a  separate  discussion  outside  the  evaluation,  the 
program  office  self-scored  the  SME  EFFECTIVENESS  as  0.5  and  SME  INVOLVEMENT  as  0.8.  Had 
this  information  been  included  in  the  AV-1,  the  Tier  I  Architecture  Quality  Value  subtotal  of  the 
joint  force  protection  VDEA-Score  would  have  improved  to  0.307  out  of  the  0.400  overall 
potential  points.  This  would  have  resulted  in  a  local  value  increase  from  0.718  to  0.767. 

4.1.23.  Measurement  Analysis 

With  the  baseline  scoring  complete,  analysis  was  conducted  on  each  measure  by  varying 
the  assessments  from  the  lowest  possibility  to  the  highest  possibility  value  to  observe  the  effect 
each  measurement  result  has  on  the  overall  score.  Figure  47  shows  the  measurement  analysis  for 
OV  READABILITY,  a  measure  with  a  continuous  S-curve  value  function.  The  original  assessment 
for  OV  READABILITY  was  4  out  of  5.  Therefore,  these  alternatives  were  generated  by  varying  the 
results  (i.e.,  x-axis)  for  OV  READABILITY  from  zero  to  one  in  increments  of  0.200  (1/5).  If  OV 
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READABILITY  was  maximized,  the  local  score  would  have  increased  by  0.005,  keeping  results  for 
all  other  measures  constant.  There  was  a  possible  0.074  local  change  VDEA-Score  varying  the 
results  from  zero  to  one. 


OV  Readability  Change  5 

0.723 

■  ■■■■ 

] 

Mill 

TTTTT 

Baseline  (As  Evaluated) 

0.718 

■  ■■■■ 

r 

Mill 

□ 

OV  Readability  Change  4 

0.703 

□ 

Z 

INI 

OV  Readability  Change  3 

0.669 

■  1  1  II  1 

c 

Z 

m 

OV  Readability  Change  2 

0.654 

II  1  II  1  1 

II  II  1  1 

n 

n 

OV  Readability  Change  1 

0.649 

II  INI 

z 

n 

®  Access  Control 

■  Document  Protection 

□ 

DoDAF  Compliancy 

□  OV  Readability 

□  SV  Readability 

□ 

Scale 

□  Decomposition 

□  Access 

□ 

File  Management 

■  File  Format 

□  Requirement  Traceability 

□ 

SME  Effectiveness 

□  SME  Involvement 

□  Connections 

□ 

Architecture  Redundancy 

□  Architecture  Economy  □Tool  Format 

□ 

Product  Locatability 

□  Internal  Consistency 

□  External  Consistency 

Figure  47.  OV  Readability  Measurement  Analysis  (||Vaq||) 


Figure  48  shows  the  measurement  analysis  for  SME  INVOLVEMENT,  a  measure  with  a 
discrete,  categorical  value  function.  The  five  alternatives  were  generated  by  choosing  each  result 
from  ‘No  Involvement’ (zero),  to  ‘Many  Stakeholder  SMEs  from  many  organizations ’(one).  It 
was  initially  assessed  ‘No  Involvement’,  therefore,  overall  value  can  only  increase,  depending  on 
the  extent  to  which  it  is  improved.  If  SME  INVOLVEMENT  was  to  earn  its  full  value,  it  would 
provide  a  0.037  increase  in  local  VDEA-Score,  keeping  results  of  all  other  measurements 
constant. 
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SME  Involvement  Change  5 
SME  Involvement  Change  4 
SME  Involvement  Change  3 
SME  Involvement  Change  2 
SME  Involvement  Change  1 
Baseline  (As  Evaluated) 


□  Access  Control 

□  OV  Readability 

□  Decomposition 

□  File  Format 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection 

□  SV  Readability 

□  Access 

□  Requirement  Traceability 

□  Connections 

□  Tool  Format 

□  External  Consistency 


□  DoDAF  Compliancy 
■  Scale 

□  File  Management 
®  SME  Effectiveness 

□  Architecture  Redundancy 

□  Product  Locatability 


Figure  48.  SME  Involvement  Measurement  Analysis  (||Vaq||) 


These  two  graphs  (Figures  47  and  48)  were  provided  as  an  example  of  the  analysis 
performed  similarly  for  the  remaining  eighteen  measures.  Graphs  for  these  measures  are 
presented  in  Appendix  E.  The  measurement  analysis  results  are  summarized  in  Table  11.  This 
table  lists  each  measure  followed  by  the  resulting  local  Architecture  Quality  Value  scores  for  a 
measurement  score  of  zero,  the  current  evaluated  score,  a  score  of  one,  and  the  delta  change  in 
overall  local  score  between  the  high  and  low  scores.  The  scores  in  italics  highlight  areas  of 
strength  where  the  evaluated  measure  scored  the  highest  value.  The  underlined  scores  highlight 
areas  of  weakness  where  the  evaluated  measure  scored  the  lowest  value. 
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Table  11.  Measurement  Analysis  Results  (||vAq||) 


Measure 

Low 

Current 

High 

Delta 

Access 

0.690 

0.718 

0.746 

0.056 

Product  Locatability 

0.690 

0.718 

0.718 

0.028 

Access  Control 

0.634 

0.084 

Document  Protection 

0.634 

0.084 

File  Management 

0.718 

0.718 

0.770 

0.052 

File  Format 

0.665 

0.053 

Connections 

0.698 

0.718 

0.730 

0.032 

Architecture  Redundancy 

0.685 

0.033 

Architecture  Economy 

0.685 

0.033 

OV  Readability 

0.649 

0.718 

0.723 

0.074 

SV  Readability 

0.664 

0.718 

0.738 

0.074 

Scale 

0.682 

0.718 

0.742 

0.060 

Decomposition 

0.658 

0.060 

Tool  Format 

0.688 

0.030 

DoDAF  Compliancy 

0.655 

0.718 

0.730 

0.075 

Requirement  Traceability 

0.718 

0.718 

0.768 

0.050 

Internal  Consistency 

0.694 

0.718 

0.719 

0.025 

External  Consistency 

0.694 

0.718 

0.719 

0.025 

SME  Effectiveness 

0.718 

0.718 

0.755 

0.037 

SME  Involvement 

0.718 

0.718 

0.755 

0.037 

4.2.  Alternative  Architecture  Evaluation 

Based  on  these  findings,  theoretical  architectures  were  conceived  to  provide  a 
comparison  of  VDEA-Score  improvement  if  the  corresponding  improved  products  were 
available.  These  options  were  determined  to  address  the  areas  in  need  of  the  most  improvement. 
With  the  exception  of  the  products  noted  as  changed,  all  other  measurement  values  were  the 
original  evaluated  scores.  The  architectures  considered  were: 
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1 .  Evaluated  with  full  value  for  OV  and  SV  READABILITY 


2.  Evaluated  with  full  value  for  REQUIREMENTS  TRACEABILITY  (assumed  validated 
SV-5  existed) 

3.  Evaluated  with  program  office  self-scored  SME  Input  values  (assumed  improved 
AV-1) 

4.  Evaluated  with  program  office  self-scored  SME  Input  values  and  full 
REQUIREMENTS  TRACEABILITY  value  (assumed  improved  AV-1  and  validated 
SV-5) 

Figure  49  shows  the  resulting  Tier  I  Architecture  Quality  Value  branch  improvements  in 
the  local  value  score  (|| vAQ  ||)  based  on  these  theoretical  architecture  changes.  These  results 
provide  an  idea  of  the  amount  of  improvement  in  the  VDEA-Score  that  he  program  office  may 
achieve  based  on  improvements  in  the  respective  areas.  This  insight  may  be  useful  to  prioritize 
limited  resources  to  concentrate  on  the  areas  of  greatest  improvement. 

As  shown  in  Figure  49,  the  addition  of  an  SV-5  in  addition  to  providing  greater  detail 
regarding  SME  Input  within  the  AV-1  would  increase  the  local  score  by  nearly  0.100  points. 

This  alternative  represents  score  changes  of  one  for  REQUIREMENTS  TRACEABILITY  and  of  0.5 
and  0.8  for  SME  EFFECTIVENESS  and  SME  INVOLVEMENT,  respectively,  based  on  the  program 
office's  self-evaluation.  It  should  also  be  noted  that  the  addition  of  an  SV-5  may  affect  the  scores 
in  other  areas  as  well,  such  as  SV  READABILITY  and  DODAF  COMPLIANCY.  However,  for 
purpose  of  showing  how  only  these  alternative  improvements  would  increase  the  Architecture 
Quality  Value  local  score,  the  other  scores  were  kept  constant  with  the  original  baseline 
evaluation. 

Adding  either  a  fully  validated  SV-5  (changing  REQUIREMENTS  TRACEABILITY  to  one 

and  leaving  both  SME  EFFECTIVENESS  and  SME  INVOLVEMENT  assessed  as  zero)  or  adding  more 

detail  regarding  SME  Input  into  the  AV-1  (changing  SME  EFFECTIVENESS  to  0.5  and  SME 
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INVOLVEMENT  to  0.8  while  leaving  REQUIREMENTS  TRACEABILITY  assessed  as  zero)  would 
raise  the  local  scores  about  0.050  (or  5  percent)  in  both  cases. 


Baseline  +  SME  Input  listed  In  AV-1  and  validated  SV-5 
0.817 

Baseline  + fully  validated  SV-5  0.768 

Baseline  +  SME  Input  listed  In  AV-1  0.767 

Baseline  +  full  valueOV/SV  Readability  0.743 

Baseline  0.718 


□  Access  Control 

□  OVReadability 

□  Decomposition 

□  File  Format 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  □  DoDAF  Compliancy 

OSV  Readability  □  Scale 

□  Access  □  FileManagement 

□  Requirement  Traceability □  SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  Product Locatability 

□  External  Consistency 


Figure  49.  Local  Architecture  Quality  Evaluation  of  Alternatives  (||Vaq||) 
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As  previously  stated,  the  Usability  branch  earned  0.742  of  its  total  Tier  II  local  value 
(74.2  percent).  Most  of  the  value  lost  was  from  the  OV  READABILITY  and  SV  READABILITY  areas. 
This  was  the  basis  for  the  other  alternative.  However,  due  to  its  relatively  lower  global  weight, 
maximizing  both  of  these  categories  only  raises  the  total  value  by  roughly  0.025  points  (2.5 
percent). 

While  most  Tier  I  Architecture  Quality  Value  branch  measures  earned  the  majority,  if  not 
all,  of  their  value,  these  alternative  VDEA-Score  results  provide  an  idea  of  the  amount  of  value 
improvement  the  program  office  may  achieve  by  acting  upon  the  recommendations.  This  insight 
may  be  useful  to  prioritize  limited  resources  to  concentrate  on  the  areas  of  greatest  improvement. 

4.3  Value  Weight  Sensitivity  Analysis 

Because  this  is  a  single  alternative  evaluation,  value  weight  sensitivity  analysis  provides 
the  opportunity  for  the  decision  maker  to  gauge  what  effect  a  value  or  measure  has  on  the  overall 
score  if  all  other  values  or  measures  were  ignored.  For  any  value  with  a  high  score,  increasing 
its  weight  increases  the  overall  score.  For  example,  Accessibility,  the  best  performing  second- 
tier  value,  earned  0.888  of  its  total  potential  value.  At  its  current  weight  of  0.250,  the  overall 
Tier  I  Architecture  Quality  Values  branch  local  score  is  0.718  (||Vaq  ||  =  0.287/0.400  =  0.718). 
Figure  50  supports  the  notion  that  if  the  weight  placed  on  this  value  is  increased,  the  overall 
score  increases  because  this  value  performed  well.  If  the  weight  was  increased  to  one,  therefore 
eliminating  the  other  second-tier  values,  the  graph  shows  the  overall  local  value  at  0.888. 
Likewise,  if  the  weight  was  lowered  from  0.250  to  zero,  thereby  eliminating  it  as  a  second-tier 
value,  the  overall  local  score  decreases  to  0.661.  Given  the  baseline  evaluation,  if  the  decision 
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maker  increases  Accessibility's  weight,  the  largest  positive  impact  on  the  overall  Architecture 
Quality  Value  score  occurs. 


On  the  other  hand,  if  a  value  scored  low,  increasing  its  weight  would  decrease  the  overall 
score.  Accountability,  the  worst  performing  second-tier  value,  earned  0.439  of  its  potential  local 
value.  At  its  current  weight  of  0.250,  the  local  Architecture  Quality  Values  score  is  0.718,  as 
displayed  in  Figure  51.  If  the  weight  increased  to  one,  basically  eliminating  the  other  three 
second-tier  values,  the  local  Architecture  Quality  Values  score  drops  to  0.439.  If  its  weight  was 
dropped  to  zero,  the  overall  local  score  rises  from  0.718  to  0.81 1.  Given  the  baseline  evaluation, 
if  the  decision  maker  increases  Accessibility's  weight,  the  largest  negative  impact  on  the  overall 
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Architecture  Quality  Value  score  occurs.  Likewise,  decreasing  Accessibility’s  weight  would 
provide  an  Architecture  Quality  Value  score  increase. 

Given  that  both  Usability  and  Modifiability  values  scored  high,  increasing  the  weight 
increases  the  overall  score.  Usability  as  shown  in  Figure  52  had  only  slight  score  changes  with 
only  a  0.050  change  in  local  Tier  I  Architecture  Quality  Value  score  between  a  zero  weight  and 
full  weight.  In  comparison  to  the  steeper  slopes  of  the  other  Tier  II  value's  sensitivity  lines,  this 
indicates  Usability  is  approximately  insensitive  to  changes  in  weight.  Therefore  the  decision 
maker  would  see  very  little  score  change  regardless  of  changes  in  Usability's  weight. 
Modifiability  as  shown  in  Figure  53  had  a  larger  change  of  0. 120  in  local  Tier  I  Architecture 
Quality  Value  score  between  a  zero  weight  and  full  weight.  This  means  the  decision  maker 
would  achieve  a  higher  score  with  an  increased  Modifiability  weight,  but  the  gain  is  not  as  large 
as  is  possible  with  Accessibility. 
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Figure  52.  Usability  Weight  Sensitivity  Analysis  for  ||Vaq|| 
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Sensitivity  Analysis  for  Modifiability 
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Figure  53.  Modifiability  Weight  Sensitivity  Analysis  for  ||vAq|| 


Tables  15  through  18  in  Appendix  F  summarize  the  weight  sensitivity  analysis  results  for 
each  of  the  values  and  measures  in  numerical  format  showing  the  maximum  positive  and 
negative  change  in  local  Tier  I  Architecture  Quality  Value  branch  VDEA-Score  as  individual 
weights  are  changed.  Table  12  also  summarizes  these  results  showing  the  values  which  had 
positive  (+A||vaq||),  negative  (— A||v4q||),  or  no  effect  (AHit^H  <  0.050)  on  the  score  with 
increased  weight.  In  general,  should  the  joint  force  protection  decision  maker  wish  to  increase 
the  Tier  I  Architecture  Quality  Value  branch  VDEA-Score  by  changing  the  assigned  weights,  the 
values  listed  with  a  positive  effect  in  Table  12  may  be  increased  in  weight  to  accomplish  this 
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goal.  Specifically,  increasing  the  Accessibility  weight  (most  positive  contributor)  while 
decreasing  the  Accountability  weight  (most  negative  contributor)  would  yield  the  largest  score 
increase. 


Table  12.  Value  Weight  Sensitivity  Effect  on  ||vAq|| 


Value 

Positive 

Effect 

No 

Effect 

Negative 

Effect 

Accessibility 

X 

Subscribability 

X 

Controllability 

X 

Protectability 

X 

Usability 

X 

Longevity 

X 

Understandability 

X 

Simplicity 

X 

Readability 

X 

Modifiability 

X 

Scalability 

X 

Tailorability 

X 

Evolvability 

X 

Accountability 

X 

Compliancy 

X 

Traceability 

X 

Consistency 

X 

SME  Input 

X 

Sensitivity  analysis  was  also  performed  on  the  proposed  alternatives  discussed  in  the 
previous  section.  In  a  situation  where  the  alternatives  vary  significantly  from  each  other, 
sensitivity  analysis  shows  how  changes  in  the  weights  affect  the  ranking  of  alternatives.  This 
allows  the  decision  maker  the  opportunity  to  see  which  alternatives  provide  the  most  value  either 
by  adjusting  the  weights  or  keeping  the  weights  as  assigned.  While  the  generated  alternatives 
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increase  score,  the  weight  sensitivity  results  for  each  of  the  alternatives  vary  in  only  a  couple  of 


areas. 

The  sensitivity  analysis  for  the  Tier  II  Usability  branch  (Figure  54)  demonstrates  that 
both  of  the  alternatives  with  the  SV-5  addition  decrease  in  overall  score  if  Usability's,  Tier  II 
local  weight  is  increased  from  0.35.  This  is  due  to  the  fact  that  these  alternatives  have  higher 
scores  for  Accountability.  If  Usability's  weight  is  increased,  the  weight  for  Accountability,  as 
well  as  Accessibility  and  Modifiability,  decreases  proportionally,  thereby  making  the  value 
earned  in  those  areas  less  important.  Three  of  the  four  alternatives  converge  at  the  same  point 
when  the  value  is  increased  to  one  because,  with  the  exception  of  the  alternative  with  full  OV  and 
SV  READABILITY  added  to  the  baseline  evaluation,  the  value  earned  under  Usability  is  identical 
for  each.  The  alternative  with  full  OV  and  SV  READABILITY  has  a  higher  value  score  when  the 
Tier  II  Usability  value  is  increased  to  one  because  both  OV  and  SV  READABILITY  measures  are 
captured  within  the  Tier  III  Understandability  branch.  This  is  the  only  alternative  with  a  higher 
Usability  score  compared  to  the  baseline  as  evaluated.  Note  that  the  alternative  "Baseline  + 

SME  Input  listed  in  AV-1"  approximately  equals  the  "Baseline  +  fully  validated  SV-5" 
alternative.  Therefore,  this  SV-5  alternative  was  eliminated  from  the  following  analysis  charts 
because  these  two  lines  would  overlap. 
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Baseline  +  SME  Input  listed  In 
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Figure  54.  Usability  Weight  Sensitivity  Analysis  (Alternatives) 


The  sensitivity  analysis  for  Accountability  (Figure  55)  shows  that  the  alternative  with 

SME  Input  added  to  the  AV-1  as  well  as  a  fully  validated  SV-5  behaves  conversely  for  this 

measure  as  its  weight  is  increased.  This  is  due  to  Accountability's  increasing  total  value  if  these 

two  improvements  are  made.  As  noted  previously,  this  area  scored  low  for  the  baseline.  Thus, 

increasing  Accountability's  weight  decreases  the  overall  score.  The  alternative  with  appropriate 

SME  information  added  to  the  AV-1  decreases  as  well  when  the  weight  is  increased.  In  this 

case,  the  Accountability  value  earned  is  not  enough  of  an  improvement  (the  value  gap  is  still 

present  for  Traceability)  to  make  a  significant  difference.  Finally,  the  analyses  for  Accessibility 

(Figure  56)  and  Modifiability  (Figure  57)  behave  similarly  in  the  sense  that  all  of  the  alternatives 

have  increasing  slopes.  This  is  due  to  the  initial  high  value  earned  in  both  of  these  branches. 

Because  no  alternatives  produced  any  extra  value  earned  under  these  two  branches  (they  all 
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scored  exactly  the  same  for  both  Accessibility  and  Modifiability ),  they  all  converge  at  the  same 
point  if  the  weight  is  increased  to  one  for  both  Tier  II  values.  In  summary,  recommendation 
rankings  are  insensitive  to  Tier  II  value  weight  adjustments  making  them  robust. 


Figure  55.  Accountability  Weight  Sensitivity  Analysis  (Alternatives) 
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Figure  56.  Accessibility  Weight  Sensitivity  Analysis  (Alternatives) 


Figure  57.  Modifiability  Weight  Sensitivity  Analysis  (Alternatives) 
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4.4.  Complete  Joint  Force  Protection  VDEA-Score 


The  complete  VDEA-Score  combines  the  System  Effectiveness  Value  (Vse)  branch  with 
the  Architecture  Quality  Value  (Vaq)  branch.  Through  similar  analysis,  as  previously  presented 

in  this  chapter,  Mills  (2009)  determined  the  System  Effectiveness  Value  branch  earned  0.248  out 
of  0.600  for  a  41.3  percent  local  value.  Therefore,  the  combined  joint  force  protection  VDEA- 
Score  was  v(x)  =  Vse(x)  +  Vaq(x)  =  0.248  +  0.287  =  0.535.  This  combined  score  is  useful  for 
noting  areas  of  improvement  and  may  serve  as  the  baseline  measure  for  future  architecture 
iterations. 

4.5.  Additional  Model  Evaluation:  IRSS 

The  focus  of  this  specific  thesis  was  a  VDEA-Score  model  for  evaluating  architecture 
products.  It  was  understood  that  the  Tier  I  System  Effectiveness  Value  branch  with  its  more 
specific  focus  on  joint  force  protection  may  require  modification  from  system  to  system. 
However,  it  is  hoped  that  the  Architecture  Quality  Values  branch  is  more  universal  even  down  to 
the  measurement  level.  To  test  this,  the  authors  preliminarily  validated  the  effectiveness  of  the 
Architecture  Quality  Values  hierarchy  using  the  IRSS  architecture.  The  results  for  the  IRSS 
analysis  are  provided  in  this  section. 

4.5.1.  IRSS  Architecture  Quality  Branch  VDEA-Score  Measure  Results 

As  mentioned  in  previous  analysis,  the  primary  source  for  evaluating  the  ACCESS 
measure  is  the  AV-1,  and  like  the  joint  force  protection  evaluation,  potentially  valuable 
information  was  missing  from  this  product.  The  evaluator  found  no  mention  of  repository  use. 
However,  the  products  were  available  in  the  Air  Force  Architecture  Repository  on  the  Air  Force 
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Knowledge  website,  thus  providing  immediate  access  for  those  with  Air  Force  Portal  access. 
Therefore  ACCESS  was  evaluated  categorically  as  "<  5  minutes"  resulting  in  a  value  score  of  one. 

As  mentioned  in  the  previous  measure,  no  pertinent  description  in  the  AV-1  was  available 
for  PRODUCT  LOCATABILITY,  ACCESS  CONTROL,  DOCUMENT  PROTECTION,  or  FILE 
MANAGEMENT.  Therefore,  the  proxy  evaluation  of  the  Air  Force  repository  was  used.  This 
resulted  in  the  following  categorical  evaluations:  PRODUCT  LOCATABILITY  evaluated  as 
"Locatable  in  <  5  minutes;"  ACCESS  CONTROL  evaluated  as  “Appropriate  Control;"  DOCUMENT 
PROTECTION  evaluated  as  "Products  Controlled;"  and  FILE  MANAGEMENT  evaluated  as  "System 
Exists,  all  products  maintained."  These  categories  each  resulted  in  value  scores  of  one. 

The  AV-1  discussed  the  tools  used  for  development.  These  tools  were  Microsoft  Office 
related  with  all  products  provided  in  those  formats.  The  evaluator  therefore  determined  the  FILE 
FORMAT  category  of  "General  File  Format"  applied  resulting  in  a  value  score  of  one.  Regarding 
CONNECTIONS,  all  products  were  presented  in  a  high-level,  simplistic  fashion.  The  links 
between  entities  were  easy-to-follow  and  well-organized.  This  resulted  in  the  evaluation  of  "15 
of  15"  products  comply  with  the  value  score  of  one.  For  ARCHITECTURE  REDUNDANCY  and 
ARCHITECTURE  ECONOMY,  the  evaluator  found  no  unnecessary  duplication  of  information  and 
no  need  to  consolidate  entities  or  activities  within  the  products.  Therefore,  ARCHITECTURE 
REDUNDANCY  and  ARCHITECTURE  ECONOMY  were  respectively  evaluated  categorically  as  "  1  in 
>  500"  and  "None  found."  These  categories  correspond  to  value  scores  of  one  for  each. 

Reviewing  the  OV  and  SV  products  for  readability,  the  evaluators  rated  the  six  OV 
products  as  easy  to  read.  Thus,  "6  out  of  6"  was  assessed  for  OV  READABILITY  resulting  in  a 
value  score  of  1.  The  SVs,  as  a  whole,  were  presented  in  a  very  easy  to  read,  almost  simplistic 
fashion.  However,  the  evaluator  determined  that  the  SV-6  was  not  intuitive  enough  as  to 
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determine  which  OV-3  events  were  being  described.  Therefore,  SV  READABILITY  was  assessed 
"5  out  of  6"  leading  to  a  value  score  of  0.950. 

As  with  the  joint  force  protection  architecture,  the  SCALE  measure  was  applied  to  all 
available  IRSS  products  to  determine  if  doubling  the  number  of  nodes  would  greatly  increase  the 
complexity.  Even  though  this  measure  is  a  fairly  subjective  assessment,  several  instances  in  the 
documentation  specifically  addressed  the  need  and  ability  to  expand  significantly.  This  provided 
extra  confidence  to  the  evaluators  who  determined  categorically  "All"  of  the  products  were 
scalable  resulting  in  the  value  score  of  one. 

The  evaluators  reviewed  the  IRSS  OV-5  for  the  DECOMPOSITION  measure.  The  OV-5 
product  had  up  to  five  levels  of  decomposition.  Thus  the  DECOMPOSITION  measure  was 
determined  to  be  categorically  "3+"  resulting  in  a  value  score  of  one. 

According  to  the  AV-1,  the  tools  used  for  the  IRSS  architecture  development  are  all 
Microsoft  Office  based.  Because  many  of  these  allow  inputs  to  be  carried  throughout  the  instant 
view  (e.g.  find  and  replace  in  Microsoft  Word)  but  not  to  others,  the  evaluator  assessed  TOOL 
FORMAT  categorically  as  "Input  gets  reflected  in  instant  view  but  not  others."  This  category 
resulted  in  a  value  score  of  0.600. 

The  Accountability  value  category  was  evaluated  last.  As  a  whole,  every  product 
appeared  in  compliance  with  DoDAF  standards.  Therefore,  the  evaluators  assessed  the  “15  out 
of  15”  DODAF  COMPLIANCY  measure  resulting  in  a  value  score  of  one.  A  complete  SV-5  was 
present  with  all  of  the  requirements  being  met  by  specific  activities  or  functions.  Therefore,  the 
REQUIREMENTS  TRACEABILITY  measure  received  a  “100%”  assessment  resulting  in  a  value 
score  of  one.  Each  entity,  function,  and  need  line  were  examined  within  every  product  for  any 
internal  inconsistencies.  Finding  none,  the  evaluator  assessed  "15  out  of  15"  for  INTERNAL 
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CONSISTENCY  resulting  in  a  corresponding  value  score  of  one.  Each  individual  product  was  then 
compared  in  relation  to  every  other  product  for  any  external  inconsistencies.  Again  the  evaluator 
found  none,  thus  assessing  "15  out  of  15"  for  EXTERNAL  CONSISTENCY  resulting  in  a 
corresponding  value  score  of  one. 

By  simply  examining  the  AV-1,  no  knowledge  of  how  effective  SMEs  were  in  developing 
the  IRSS  architecture  was  found.  Specifically,  no  mention  of  the  use  of  SMEs  or  their 
experience  was  located  so  no  knowledge  of  the  developmental  team  was  captured.  Therefore, 
the  evaluator  assessed  "No  information  provided"  for  SME  EFFECTIVENESS  resulting  in  a  value 
score  of  zero. 

For  the  SME  INVOLVEMENT  measure,  the  AV-1  was  again  reviewed.  As  a  single  service 
project,  the  categories  were  tailored  to  major  commands  (MAJCOMs)  for  IRSS  versus  services 
for  the  force  protection  evaluation.  While  not  specifically  mentioned  as  SMEs,  the  document 
does  describe  the  IRSS  Requirements  Review  Board  with  specific  membership  of  14  different 
Air  Force  MAJCOM-level  organizations  who  were  also  identified  as  users.  Therefore,  the 
evaluator  made  the  assumption  these  organizations  would  provide  SME-type  input  but  did  not 
give  credit  for  multiple  SMEs  from  the  multiple  organizations  because  that  could  not  be  deduced. 
As  such,  SME  INVOLVEMENT  was  assessed  as  "4+  organizations"  with  the  resulting  value  score 
of  0.8. 

4.5.2.  IRSS  Architecture  Quality  VDEA-Score  Summary 

The  final  step  in  providing  a  single  Architecture  Quality  Value  VDEA-Score  requires  the 
summation  of  each  of  the  individual  value  scores  according  to  their  respective  global  weights 
using  to  the  general  additive  value  function  of  v(x)  =  £f=1  Ajfj(Xj).  Table  13  is  provided  as  a 
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summary  of  these  individual  scores  with  the  resulting  score.  The  graph  shown  in  Figure  58 
shows  the  detail  of  these  measure  scores  in  comparison  to  the  full  potential  value.  This  0.378 
score  out  of  0.400  possible  represents  approximately  95  percent  of  its  total  potential  VDEA- 
Score  for  Tier  I  Architecture  Quality  Value. 


Table  13.  IRSS  Architecture  Quality  Value  Scoring 


Measure 

Assessment 

Weight 

k 

Value 

Score 

Vi(Xi) 

Product 

Access 

Proxy  Eval  of  Repository: 

Access  <  5min 

0.022 

1.000 

0.022 

Product  Locatability 

Proxy  Eval  of  Reposity: 
Locatable  <  5min 

0.011 

1.000 

0.011 

Access  Control 

Proxy  Eval  of  Repository: 
Appropriate  Control 

0.033 

1.000 

0.033 

Document  Protection 

Proxy  Eval  of  Repository: 
Write-Protected 

0.033 

1.000 

0.033 

File  Management 

Proxy  Eval  of  Repository: 

System  exists 

0.021 

1.000 

0.021 

File  Format 

General  File  Formats 

0.021 

1.000 

0.021 

Connections 

15  out  of  15 

0.013 

1.000 

0.013 

Architecture 

Redundancy 

0  redundancy  instances  found 

0.013 

1.000 

0.013 

Architecture  Economy 

No  instances  of  possible  consolidation 

0.013 

1.000 

0.013 

OV  Readability 

6  out  of  6 

0.030 

SV  Readability 

5  out  of  6 

0.0285 

Scale 

All  Scalable  2x 

0.024 

1.000 

0.024 

Decomposition 

3+  levels 

0.024 

Tool  Format 

Input  carries  instant  view 

0.0072 

DoDAF  Compliancy 

15  out  of  15 

0.030 

Req’t  Traceability 

15  out  of  15 

0.022 

Internal  Consistency 

15  out  of  15 

0.010 

External  Consistency 

15  out  of  15 

1.000 

0.010 

SME  Effectiveness 

No  info  provided 

0.000 

0.000 

SME  Involvement 

Multiple  Organizations 

0.015 

0.800 

0.012 

n 


^4q0*0  ^  ^  ^ iVi  Ol)  0.378 

_ i=i _ 
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Figure  58.  IRSS  Evaluated  ||Vaq||  over  Potential  ||Vaq|| 


As  noted  in  the  joint  force  protection  evaluation,  SME  Input  was  again  identified  as  the 
key  area  of  improvement.  It  is  also  quite  likely  that  in  practice  the  program  has  significant  SME 
support  which  would  increase  their  score  had  it  been  identified  in  their  AV- 1 .  Similar  to  the 
joint  force  protection  evaluation,  more  detail  in  the  AV-1  regarding  the  use  of  the  official  Air 
Force  repository  would  have  provided  more  direct  measurement  for  the  three  Accessibility  value 
measures  and  Longevity.  Overall,  this  higher  VDEA-Score  reflects  the  architecture's  maturity 
(final  product  as  opposed  to  the  draft  joint  force  protection  architecture)  and  narrower  program 
focus  (multi-user  database  for  only  the  Air  Force  as  opposed  to  the  joint  force  protection 
architecture's  joint  nature  encompassing  many  disparate  subsystems). 


128 


In  the  course  of  this  second  case  study,  the  SME  Involvement  measure  was  highlighted  as 
requiring  modification.  With  the  initial  development  focused  on  the  joint  force  protection 
architecture,  the  original  SME  Involvement  measure  was  defined  in  terms  of  number  of  services 
involved.  With  the  IRSS  architecture,  the  single-service  nature  of  the  architecture  required  a 
change  in  definition  to  number  of  stakeholder  organizations  making  the  measure  more  widely 
applicable. 
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V.  Conclusions  and  Recommendations 


This  chapter  summarizes  the  research  and  findings  of  the  Value-Driven  Enterprise 
Architecture  Score  (VDEA-Score)  analysis  for  enterprise  architecture  evaluation  using  weighted 
stakeholder  value  categories.  The  answers  to  the  initial  research  questions  are  summarized 
followed  by  recommendations  to  the  sponsor  for  architecture  improvements.  Finally,  the 
strengths  and  weaknesses  of  the  model  and  suggested  future  research  are  also  presented. 

5.1.  Answers  to  Research  Questions 

Early  in  this  thesis,  four  major  research  questions  were  posed.  These  were: 

1 .  What  are  the  “best”  methods  to  evaluate  and  measure  the  overall  quality  of  an 
architecture? 

2.  What  are  the  major  categories  and  sub-categories  that  should  be  considered  when 
evaluating  an  architecture? 

3.  What  are  the  major  categories  and  sub-categories  that  should  be  considered  when 
evaluating  force  protection  processes? 

4.  How  do  these  categories  and  sub-categories  rank  in  terms  of  importance? 

5.  How  well  does  current  joint  force  protection  architecture  meet  the  weighted 
values  of  the  force  protection  community? 

While  a  variety  of  approaches  to  evaluate  and  measure  architecture  quality  exist,  no  single, 

“best”  approach  was  found.  The  research  team  found  an  architecture  can  be  viewed  as  an 
incumbent  solution  to  a  decision  situation.  Using  principles  from  Value-Focused  Thinking 
(VFT)  provided  the  optimal  foundation  for  development  of  the  VDEA-Score  to  evaluate  and 
measure  the  overall  quality  of  this  architecture  solution.  Through  extensive  research,  a 
comprehensive  list  of  ‘-ilities’  was  developed.  This  listing  was  further  grouped  into  categories 
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assessed  as  valuable  to  the  project  goals.  With  input  and  validation  from  the  decision  maker, 
these  categories  were  transformed  into  sets  of  attributes  deemed  most  valuable  to  the  decision 
maker  to  evaluate  both  architecture  quality  and  force  protection  processes.  These  resulting  two 
sets  formed  the  two  major  branches  of  the  overall  value  hierarchy:  System  Effectiveness  Values 
and  Architecture  Quality  Values.  One  or  more  measures  associated  with  each  of  the  lowest-tier 
values  were  developed  to  enable  evaluation.  This  answered  the  aforementioned  research 
questions  two  and  three. 

Because  these  values  were  not  equally  important,  weights  were  assigned  in  terms  of 
importance  to  each  value  and  measure  contained  within  the  hierarchy  in  answer  to  question  four. 
These  weights  allowed  computation  of  an  overall  score  that  acts  as  “value  earned,”  as  opposed  to 
acting  as  a  “grade.”  This  score  evaluated  both  the  quality  of  the  instantiated  system  being 
represented  and  its  ability  to  perform  its  stated  mission  (system  effectiveness)  and  the  intrinsic 
quality  of  the  products  in  terms  of  documentation  standards  and  desired  attributes  (architecture 
quality). 

To  answer  question  five,  the  resulting  VDEA-Score  model  was  used  to  evaluate  the  joint 
force  protection  architecture.  The  overall  joint  force  protection  VDEA-Score  was  assessed  to  be 
0.535  out  of  the  potential  1 .000.  In  other  terms,  53.5  percent  of  the  total  value  to  this  point  was 
earned.  The  primary  focus  of  this  thesis  was  on  the  Architecture  Quality  Values  branch,  which 
earned  0.287  out  of  a  possible  0.400  points,  or  71.8  percent  of  its  possible  value.  Due  to  the  fact 
that  the  joint  force  protection  architecture  is  still  in  early  draft  stages,  areas  for  improvement 
were  highlighted  through  this  evaluation.  This  score  further  acts  as  a  reference  point  for  the 
decision  maker  to  use  to  compare  future  architecture  iterations.  Specific  recommendations  to 
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gain  more  value  in  regards  to  the  joint  force  protection  architecture  quality  follow  in  the  next 
section. 

5.2.  Recommendations 

Intended  to  aid  the  decision  maker  in  determining  which  steps  to  take  next, 
recommendations  were  developed  based  on  the  overall  score  as  well  as  the  deterministic  and 
sensitivity  analysis.  The  majority  of  measures  within  the  Architecture  Quality  Values  branch 
were  evaluated  using  an  aggregate  of  available  views.  There  were  only  three  views  that  served  as 
the  single  source  for  evaluating  any  given  measure:  the  AV-1,  SV-5  and  OV-5. 

The  AV-1,  in  particular,  was  the  sole  source  for  evaluating  9  of  the  20  measures. 
Additions  to  the  AV-1,  primarily  relating  to  detailed  information  pertaining  to  SME  Input,  could 
provide  an  increase  of  0.049  of  the  local  or  normalized  Tier  I  Architecture  Quality  Value  VDEA- 
Score  from  0.718  to  0.767  points.  This  assumed  the  program  office  self-evaluated  scores  of  0.5 
and  0.8  for  SME  EFFECTIVENESS  and  SME  INVOLVEMENT,  respectively  (if  these  two  measures 
were  maximized,  the  jump  would  be  even  higher).  Improving  these  two  measure  scores  would 
provide  significant  additional  earned  value  to  the  Tier  II  value  component  in  need  of  most  work: 
Accountability  (the  lowest  scoring  of  the  four  second-tier  values).  Sensitivity  analysis  also 
confirms  this  low  score  would  cause  the  largest  loss  in  value  if  the  decision  maker  decided  to 
increase  Accountability's,  weight. 

Although  not  providing  any  score  increase,  other  additions  to  the  AV-1  may  improve 
direct  evaluation  of  the  architecture.  Information  related  to  steps  taken  to  control  access  and 
protection  of  the  architecture  products  as  well  as  methods  of  development  for  electronic  products 
could  be  placed  in  the  AV-1  to  ease  direct  and  indirect  evaluation  of  Tier  II  value  Accessibility. 
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Another  cause  for  value  lost  in  Accountability  was  due  to  zero  value  earned  in 
Traceability.  This  was  directly  related  to  the  absence  of  the  Operational  Activity  to  Systems 
Function  Matrix  (SV-5)— the  sole  source  for  the  REQUIREMENTS  TRACEABILITY  measure.  The 
SV-5  is  a  good  way  to  show  which  systems  are  performing  certain  functions,  thus  allowing 
traceability  from  operational  requirements  to  system  functions.  Like  the  previous 
recommendation,  this  also  provides  a  0.050  increase  in  the  normalized  Tier  I  Architecture 
Quality  Value  VDEA-Score  from  0.718  to  0.768. 

Merely  improving  the  AV-1  or  creating  the  SV-5  would  theoretically  provide  value. 
However,  operational  requirements  are  not  listed  in  the  AV-1.  If  the  evaluator  is  not  aware  of 
the  operational  requirements,  the  SV-5  provides  nothing  regarding  requirements  traceability. 
Therefore,  the  authors  recommend  updating  the  AV-1  and  completing  the  SV-5  starting  with 
operational  requirements  documentation  which  provides  a  nearly  0.100  increase  of  the 
normalized  total  Architecture  Quality  Value  from  0.718  to  0.817.  Further,  the  creation  of  the 
SV-5  may  provide  additional  increase  or  decrease  in  value  for  other  measures  such  as  DODAF 
COMPLIANCY  and  SV  READABILITY  which  rely  on  the  ratio  of  products  in  accordance  to  the  total 
number  of  products.  These  possibilities  were  not  accounted  for  when  conducting  the  analysis. 
However,  assuming  the  best  case  that  the  SV-5  would  be  readable,  consistent  with  the  other 
views,  and  compliant  with  DoDAF  standards,  the  normalized  total  Architecture  Quality  Value 
would  increase  to  0.826. 

Correcting  minor  issues  related  to  DODAF  COMPLIANCY  provided  less  of  an  increase  in 
value  earned  but  should  be  considered.  Per  DoDAF  Vol  II,  version  1.5  (2008),  the  SV-6  needs 
more  description  of  the  data,  and  the  OV-2  needs  the  information  being  exchanged  among 
entities.  Even  though  this  measure  earned  0.830  of  its  local  potential  value,  it  is  one  of  the 
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highest  weighted  measures.  Improving  this  score  would  increase  the  contribution  Accountability 
gives  to  the  Tier  l  Architecture  Quality  Value  score. 

To  increase  value  earned  for  Usability,  the  problems  with  merged  and  unlabeled 
connections  within  the  SV-1  and  SV-4  as  well  as  the  lack  of  a  file  management  system  need  to 
be  resolved.  Fixing  the  SV-1  and  SV-4  needlines  would  improve  the  sub-tier  CONNECTIONS 
measurement.  Implementing  an  official  file  management  system  along  with  documenting  it  in 
the  AV-1  would  improve  the  FILE  MANAGEMENT  measurement. 

5.3.  Model  Strengths 

By  starting  with  a  comprehensive  list  of  "ilities,"  the  Value-Focused  Thinking  approach 
behind  the  VDEA-Score  methodology  was  beneficial  to  transform  these  "ilities"  into  an 
organized  and  simple  value  hierarchy  useful  to  multiple  enterprise  architecture  evaluations.  In 
the  case  of  the  Architecture  Quality  Values  branch,  the  values,  measures,  and  value  functions 
represent  aspects  important  to  any  architecture.  This  branch  was  intentionally  separated  to 
enable  its  reuse  to  apply  to  any  system's  architecture.  Thus,  the  VDEA-Score  Tier  I  Architecture 
Quality  Values  is  very  portable.  The  value  hierarchy  may  also  be  a  good  starting  point  for 
measuring  System  Effectiveness  Values  but  will  likely  need  to  be  revised  at  Tier  III. 

Because  all  values  are  not  equally  important,  each  value  has  an  associated  weight.  Given 
that  different  decision  makers  likely  have  different  perspectives  on  each  value's  importance, 
these  weights  can  easily  be  tailored.  This  flexibility  further  enhances  the  model's  reusability  for 
any  system  architecture. 

As  with  any  evaluation  process,  repeatability  is  important  to  ensure  credibility  of  results. 
The  measures  for  each  of  the  Architecture  Quality  Values  were  designed  and  defined  to  enable 
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different  evaluators  to  apply  this  model  and  determine  the  same  results.  This  enhances  the 
credibility  as  well  as  the  usefulness  by  not  requiring  a  specialized  consultant  to  conduct  the 
evaluation. 

Through  the  application  to  the  two  systems  presented  in  this  thesis,  the  Tier  I 
Architecture  Quality  Value  branch  of  the  VDEA-Score’s  repeatability,  tailorability,  and 
portability  were  demonstrated.  Further,  this  model  was  useful  in  identifying  the  architectural 
areas  of  strength  and  weakness  to  our  sponsor  to  enable  product  improvement.  The  separate 
analysis  of  IRSS  provided  initial  indication  that  the  evaluation  tool  can  be  applied  to  a  variety  of 
systems  at  different  levels  of  acquisition  development. 

5.4.  Model  Weaknesses 

While  this  model's  usefulness  was  verified  across  two  systems,  the  sample  set  of  only  two 
architectures  does  not  provide  sufficient  validation.  Additionally,  only  the  joint  force  protection 
decision  maker  was  involved.  Therefore,  the  actual  tailorability  of  applying  different  weights 
according  to  a  different  decision  maker  was  not  tested.  The  repeatability  of  the  evaluation  was 
also  not  demonstrated  in  this  effort  because  only  the  authors  served  as  evaluators  with  the 
exception  of  the  OV  READABILITY  measure.  As  was  discovered  in  the  OV  READABILITY 
evaluation,  a  tradeoff  in  values  (e.g.,  the  larger  amount  of  detail  required  to  make  the  OV-5 
useful  for  the  complex  joint  force  protection  architecture  while  sacrificing  readability)  may  also 
preclude  achieving  a  full  value  VDEA-Score. 

Even  though  sufficient  measures  were  developed  through  this  effort,  the  significant 
portion  of  qualitative  and  categorical  measures  is  a  known  weakness  of  this  model.  More  direct 
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measures  may  be  possible  and  tailorable  to  specific  programs.  In  particular,  the  following 
measures  have  specific  weaknesses  identified  by  the  authors. 

o  SCALABILITY:  This  was  a  very  subjective  assessment  of  a  product’s  ability  to 
double  in  size  without  significantly  increasing  complexity. 

o  ARCHITECTURE  ECONOMY:  This  was  a  very  subjective  binary  assessment  of  any 
multiple  steps  unnecessarily  used  to  represent  the  same  activity.  Without 
interviewing  the  architect,  it  was  difficult  to  determine  solely  from  the  products  if 
any  instances  were  truly  unnecessary  or  were  purposely  described  in  multiple 
steps. 

o  SME  INVOLVEMENT:  In  the  case  of  joint  force  protection,  a  larger  number  of 
SMEs  involved  was  termed  beneficial.  However,  more  is  not  always  better  as 
more  individuals  may  also  mean  more  differing  perspectives  requiring  more  work 
to  reconcile  differences. 

o  FILE  MANAGEMENT:  This  measure  was  defined  to  allow  a  proxy  evaluation  of  an 
official  architecture  repository  (e.g.,  DARS)  to  score  full  value  if  not  described  in 
the  products.  Because  only  one  product  version  is  kept  in  the  repository,  this 
measure  may  not  completely  capture  the  usefulness  of  an  actual  file  management 
plan.  Thus,  access  to  drafts  for  coordination  or  historical  versions  is  not  possible. 

O  ACCESS  CONTROL,  FILE  FORMAT,  CONNECTIONS,  ARCHITECTURE  ECONOMY, 
ARCHITECTURE  REDUNDANCY,  OV  READABILITY,  SV  READABILITY,  SCALE, 

TOOL  FORMAT,  and  SME  EFFECTIVENESS:  These  are  constructed,  proxy 
measures.  This  represents  half  of  the  total  Architecture  Quality  Value  measures 
which  conflicts  with  the  goal  to  minimize  this  type  of  measure  in  favor  of  natural, 
direct  measures. 


It  is  also  important  to  note  the  VDEA-Score  model  is  focused  on  the  visualization  aspects 
(products  and  views)  of  the  DoDAF.  As  the  DoDAF  transitions  from  this  product-centric 
approach  to  a  data-centric  one,  the  VDEA-Score  measures  may  need  refining.  In  particular, 
addressing  the  Core  Architecture  Data  Model  (CADM)  and  the  data  itself  versus  its  visualization 
may  be  required.  The  authors  also  note  this  model  is  a  descriptive  evaluation  of  a  program’s 
architecture.  For  insight  into  a  potential  prescriptive  approach  for  programs  with  limited 
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architectural  development  resources  to  develop  the  most  effective  architecture  products,  the 
reader  should  refer  to  the  third  thesis  associated  with  this  effort  (Osgood,  2009). 

5.5.  Future  Research 

To  address  these  identified  weaknesses,  the  authors  recommend  future  research  to 
enhance  the  VDEA-Score.  Additional  application  to  other  system  architectures  is  recommended 
for  validation  of  the  VDEA-Score  model’s  applicability  to  any  system  architecture.  Both  joint 
and  single  service  (from  different  services)  system  architectures  should  be  evaluated.  These 
additional  architecture  evaluations  should  also  involve  different  decision  makers  for 
demonstration  of  the  VDEA-Score’ s  tailorability.  Because  only  the  authors  served  as  the 
evaluators,  use  of  additional  evaluators  scoring  the  same  architectures  independently  is  also 
recommended  to  confirm  the  repeatability. 

Developing  more  direct  measures  for  existing  value  components  would  likely  expand  the 
objectivity  and  quantifiability  of  the  VDEA-Score.  As  noted  by  the  high  score  for  the  IRSS 
architecture,  more  direct  measures  or  additional  discrimination  within  measure  categories  may 
provide  more  discrimination  in  the  overall  VDEA-Score.  This  would  reduce  the  likelihood  of  a 
100  percent  score  situation  which  provides  no  assistance  to  the  program  office  in  identifying 
areas  of  improvement. 

Because  many  of  the  OV  and  SV  architecture  products  involve  entities  and  needlines, 
these  may  be  interpreted  as  nodes  and  paths.  Therefore,  network  flows  (Ahuja  et  al.,  1993)  and 
graph  theory  (West,  1996)  may  be  applied  to  incorporate  the  concepts  of  shortest  path,  most 
connected  node,  cliques,  etc.  As  an  example  of  graph  theory  application,  inconsistencies  in 
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architecture  design  “could  be  looked  up  by  the  paths  length  checking  in  the  combined  graph 
which  depicts  the  structural  relationship  of  OV2  and  OV5”  (Liu,  2007). 

Additionally,  the  DoDAF  continues  to  evolve  with  the  DoD  net-centric  transformation 
and  advances  in  enabling  technologies  such  as  services  within  Service  Oriented  Architecture 
(SOA).  The  DoDAF  transition  from  a  product-centric  focus  in  DoDAF  version  1 .5  to  a  data- 
centric  focus  in  DoDAF  version  2.0  may  require  more  research  into  new  VDEA-Score  measures 
which  account  for  this  change.  As  a  potential  starting  point  for  this  research,  the  Architecture 
Verification  and  Integration  for  DoDAF  (AVID)  prototype  from  Trident  Technology  Solutions 
(Reber,  2009)  should  be  examined. 

Specific  to  the  642  ELSS,  the  authors  also  suggest  future  research.  Besides  the 
development  of  architecture,  the  program  office  receives  numerous  proposals  from  industry  for 
new  force  protection  equipment.  As  an  additional  tool  to  aid  the  program  office,  future  research 
is  suggested  building  on  the  VDEA-Score  methodology  for  the  evaluation  of  these  new  industry 
proposals  for  system  component  acquisition. 

5.6.  Conclusion 

The  VDEA-Score  methodology  demonstrated  to  the  sponsor  and  the  authors  the 
usefulness  of  this  new  tool  for  evaluating  the  quality  of  system  architecture.  It  is  important  to 
remember  the  VDEA-Score  is  not  a  "grade"  but  merely  a  tool  to  highlight  areas  of  strength  and 
illuminate  areas  for  improvement.  Overall,  this  evaluation  identified  important  areas  of 
improvement  providing  new  insight  to  the  sponsor  of  the  possible  paths  to  high  quality 
architecture  as  the  building  block  to  actual  system  development.  This  baseline  score  can  be  used 
to  compare  future  iterations  of  their  architecture.  In  addition  to  this  thesis,  the  results  of  the 


138 


VDEA-Score  research  were  captured  in  an  outbrief  to  the  642d  ELSS,  papers  accepted  for  the 
2009  Industrial  Engineering  Research  Conference  (Mills  et  al.,  2009b)  and  Conference  on 
Systems  Engineering  Research  (Cotton  et  al.,  2009),  as  well  as  the  Mills  (2009)  and  Osgood 
(2009)  theses. 
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Appendix  A.  Ilities  Table 


The  following  table  lists  the  “ilities”  considered  for  this  effort  as  determined  by  the  associated  references  and  through  brainstorming. 
The  Disposition  column  describes  which  ones  were  used  (bold  italics),  which  ones  were  covered  by  the  ones  used,  and  which  ones 
were  discarded.  Legend:  AQ=Architecture  Tier  1;  SE=System  Tier  1;  Sub=Sub-tier 


Literature  Reference 


"llity" 

cr> 

o 

o 

rsi 

nf 

TD 
( 1) 

Q_ 

1^: 

5 

Haskins,  2006 

Ross,  Hastings, 

2006 

Ross,  2006 

McManus,  2007 

Voas,  2004 

Dalgren,  2007 

Bottella,  2004 

SEI  2003 

Daniels,  2005 

Richards,  2006 

Lehto,  2005 

Ross,  Rhodes, 

2008 

Schultz,  1999 

Total  Referenced 

Disposition 

accessibility 

X 

1 

AQ1 

accountability 

X 

1 

AQ2 

accuracy 

X 

X 

2 

AQ4Sub  ICovered 

adaptability 

X 

X 

X 

X 

X 

X 

X 

7 

SE  1  Sub  3  Covered 

administrability 

X 

1 

AQ  1  Sub  3  Covered 

affordability 

X 

X 

X 

B 

SE  1  Sub  2  Covered 

agility 

X 

X 

X 

B 

SE  1  Sub  3  Covered 

analysability 

X 

1 

AQ  2  Sub  2.2  Covered 

analytic  extensibility 

X 

1 

AQ  2  Sub  2.2  Covered 

anticipation 

X 

1 

AQ  3  Sub  3  Covered 

applicability 

X 

1 

SE  1  Sub  1  Covered 

attractiveness 

X 

1 

Discarded 

auditability 

X 

1 

AQ4Sub  3  Covered 

autonomy 

X 

2 

Discarded 

availability 

X 

X 

X 

X 

X 

1 

AQ  2  Sub  1  Covered 

business  horizontalization 

X 

1 

Discarded 

capacity 

X 

1 

Discarded 

capability 

X 

1 

SE  1 

changeability 

X 

X 

X 

1 

AQ  3  Sub  2  Covered 

clarity 

X 

AQ  2  Sub  2.2  Covered 
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Wikipedia,  2009 
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Ross,  2006 
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o 

o 

dP 

1 

o 

PQ 

SEI2003 

Daniels,  2005 

Richards,  2006 

Lehto,  2005 

Ross,  Rhodes,  2008 

Schultz,  1999 

Total  Referenced 

Disposition 

co-existence 

X 

1 

SE  3  Sub  2  Covered 

communication 

0 

SE  3  Sub  1 

commonality 

X 

1 

SE  3  Sub  2  Covered 

compatibility 

X 

m 

H 

SE  3  Sub  2  Covered 

complexity 

0 

AQ  2  Sub  2.1  Covered 

compliancy 

X 

X 

2 

AQ  4  Sub  1 

compos  abiEty 

X 

1 

Discarded 

con  figurabi  lity 

X 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

1 

AQ  3  Sub  2  Covered  / 
SE  Sub  3  Covered 

consistency 

X 

1 

AQ  4  Sub  4 

constructability 

Q 

SE  1  Sub  2  Covered 

controllability 

X 

i 

AQ  1  Sub  3 

credibiEty 

X 

i 

AQ  4  Sub  3  Covered 

customizabiEty 

X 

i 

AQ  3  Sub  2  Covered 

data  integrity 

Q 

AQ  4  Sub  4  Covered 

decentralization 

m 

i 

Discarded 

degradabiEty 

X 

i 

SE  2  Sub  1 .2  Covered 

demonstrabiEty 

X 

i 

Discarded 

depen  dabi  l  i  ty 

X 

i 

SE  2  Sub  1 

deployability 

X 

i 

SE  1  Sub  2  Covered 

diagnoseabiEty 

0 

SE  2  Sub  2.2  Covered 

distributabiEty 

X 

1 

AQ  1  Sub  1  Covered 

durability 

X 

X 

111 

AQ  2  Sub  1  Covered 
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Wikipedia,  2009 

Haskins,  2006 

Ross,  Hastings,  2006 
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Dalgren,  2007 
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SEI2003 

Daniels,  2005 
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Lehto,  2005 

Ross,  Rhodes,  2008 

Schultz,  1999 
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<L> 
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o 
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Disposition 

effecti  ven  ess 

0 

System  Branch 

efficiency 

X 

X 

2 

SE  1  Sub  2  Covered 

environmental  cost 

X 

1 

SE  2  Sub  2  Covered 

evolvability 

X 

1 

AQ  3  Sub  3 

executeability 

X 

1 

SE  1  Sub  2  Covered 

extensibility 

X 

2 

AQ  3  Sub  1  Covered 

foil  safe 

X 

1 

Discarded 

fault  tolerability 

X 

X 

X 

3 

SE2  Sub  1.2  Covered 

feasibility 

0 

SE  1  Sub  2  Covered 

fidelity 

X 

1 

SE2  Sub  1.2  Covered 

flexibility 

X 

X 

X 

X 

X 

X 

X 

X 

X 

9 

SE  1  Sub  3 

fimctionality 

X 

X 

2 

SE  1  Sub  1  Covered 

integrability 

X 

1 

SE  3  Sub  2  Covered 

installability 

X 

X 

X 

3 

SE  1  Sub  2  Covered 

interchangeability 

X 

1 

SE  3  Sub  2 

Intemationalizability 

0 

SE  3  Sub  2  Covered 

i  n  teroperabili  ty 

X 

X 

X 

X 
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SE  3 

leamability 

X 

X 

X 

3 
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longevity 

0 

AQ  2  Sub  1 
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o 

PQ 

SEI 2003 

Daniels,  2005 

Richards,  2006 

o 

o 
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Ross,  Rhodes,  2008 

Schultz,  1999 

Total  Referenced 

Disposition 

maintainability 

X 

X 

X 

X 

X 

X 

B 

SE  2 

manageability 

X 

1 

AQ  1  Sub  3  Covered 

manufacturability 

X 

1 

SE  1  Sub  2  Covered 

maturity 

X 

1 

SE  1  Sub  2  Covered 

mobi  lity 

X 

1 

SE  1  Sub  3  Covered 

modifiability 

X 

1 

AQ  3 

modularity 

X 

X 

X 

B 

SE  1  Sub  3  Covered 

nomadicity 

X 

1 

Discarded 

ope  mess 

0 

AQ  1  Sub  3  Covered 

ope  rability 

X 

X 

X 

3 

SE  1  Sub  2  Covered 

Performance 

X 

X 

2 

SE  1  Sub  1  Covered 

P  erson  alizability 

o 

AQ  3  Sub  2  Covered 

par  tability 

X 

X 

X 

H 

Discarded 

practicality 

O 

SE  1  Sub  2 

precision 

X 

i 

AQ  4  Sub  1  Covered 

predictability 

X 

i 

Discarded 

produceability 

o 

SE  1  Sub  2  Covered 

profitability 

X 

Discarded 

protectability 

o 

AQ1  Sub  2 

purposefulness 

X 

1 

SE  1  Sub  1 

quality 

X 

1 

Overall  VDEA  Score 
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McManus,  2007 
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Bottella,  2004 

SEI  2003 

Daniels,  2005 

Richards,  2006 

Lehto,  2005 

Ross,  Rhodes,  2008 

Schultz,  1999 

Total  Referenced 

Disposition 

readability 

B 

AQ  2  Sub  2.2 

recoverability 

X 

B 

B 

SE  2  Sub  2.2 

redundancy 

B 

1 

SE  2  Sub  1 .2  Covered 

relevance 

X 

1 

SE  1  Sub  1  Covered 

reliability 

X 

X 

X 

X 

X 

X 

X 

X 

X 

B 

SE  2  Sub  1.2 

repairability 

X 

B 

B 

SE  2  Sub  1 . 1  Covered 

repeatability 

X 

1 

AQ  3  Sub  1  Covered 

replaceability 

B 

1 

SE  1  Sub  2  Covered 

reproducibility 

X 

1 

AQ  3  Sub  1  Covered 

resiliancy 

B 

SE  2  Sub  2 

resource  utilisation 

B 

1 

SE  1  Sub  2  Covered 

responsiveness 

X 

1 

SE  1  Sub  3  Covered 

reusability 

X 

B 

B 

AQ  3  Sub  2  Covered 

robustness 

X 

X 

X 

X 

X 

X 

X 

SE  1  Sub  1  Covered 

safety 

X 

X 

SE  1  Sub  2  Covered 

scalability 

X 

X 

X 

X 

B 

B 

X 

X 

B 

B 

AQ  3  Sub  1 

seamlessness 

X 

1 

Discarded 

securability 

X 

X 

B 

AQ  1  Sub  2  Covered 

security 

X 

X 

X 

AQ  1  Sub  2  Covered 

serviceability 

X 

X 

SE  2  Sub  1 . 1  Covered 

simplicity 

X 

B 

B 

B 

AQ  2  Sub  2.1 
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Disposition 

stability 

X 

X 

2 

SE  2  Sub  1.2  Covered 

standardization 

0 

AQ  4  Sub  1  Covered 

stakeholder  involvement 

0 

AQ  4  Sub  2  (translated 
to  SME  Input  value) 

subscribabil  ity 

0 

AQ  1  Sub  1 

supportability 

X 

1 

SE2  Sub  1.1 

survivability 

X 

X 

2 

SE  2  Sub  2.1 

suscept  ability 

0 

SE  2  Sub  2.1  Covered 

sustainability 

X 

X 

X 

3 

SE  2  Sub  1 . 1  Covered 

suitability 

X 

1 

SE  1  Sub  2  Covered 

tailorability 

X 

1 

AQ  3  Sub  2 

testability 

X 

X 

X 

X 

X 

X 

6 

Discarded 

timeliness 

X 

X 

X 

3 

Discarded 

traceability 

0 

AQ  4  Sub  3 

trainability 

0 

Discarded 

transactionality 

0 

Discarded 

understandability 

X 

X 

2 

AQ  2  Sub  2 

Upgrade  ability 

X 

1 

AQ  3  Sub  3  Covered 

usability 

X 

X 

X 

X 

4 

AQ  4 

utility 

0 

SE  1  Sub  1  Covered 

vulnerability 

X 

1 

AQ  1  Sub  2  Covered 

versatility 

X 

X 

HI 

SE  1  Sub  3  Covered 

Total  in  Reference 

61 

5 

4 

~n 

6 

EE! 

B 

B 

~6~ 

8 

BE 

5 

15 

Appendix  B.  VDEA-Score  Evaluation  Sheet 


Accessibility 


Subscribability 


□ 

Access 

1 

Do  stakeholders  have  electronic 
access  to  products? 

No  means  to 

gain  access 

>  1  week  to 
gain  access 

3  days  <  access 
granted  <  1 
week 

5  mins  <  access 
granted  <  3 
days 

access  granted 
<  5  mins 

Product  Locatability 

2 

Can  stakeholders  easily  locate 
electronic  products? 

Can  not  locate 
products 

>  5  minutes  to 
locate  products 

<  5  minutes  to 
locate  products 

Protectability 

Access  Control 

3 

Are  access  control  measures 
implemented  to  appropriate 
level  of  protection? 

No  plan  or  plan 
inadequate 

Plan  exists,  but 

not 

implemented 

Appropriate 

protection 

implemented 

Controllability 

Document  Protection 

I 

Are  the  products  appropriately 
write  protected? 

No  plan  for 
write  protection 

Plan  exists,  but 

not 

implemented 

Plan  exists,  all 
products 
controlled 

□ 

Usability 


Longevity 


File  Management 

5 

Has  an  official  file  ma  na  ge  me  nt  syste  m  for 
keeping  products  been  established? 

No  offi ci  a  1  fi  1  e 

management  system 

System  exists,  but 
i  ncompl  ete/not 
ma  i  nta  i  ned 

System  exists  with  all 
p ro d u cts  and  is 
ma  i  nta  i  ned 

File  Format 

6 

To  what  degree  is  there  a  reasonable 
expectation  that  the  electronic  products  will  be 
available  in  the  future? 

No  electronic  products 
or  no  longer 
accessible 

Only  accessi  ble  with 
one  type  of 
propri  eta  ry  softwa  re 

File  format  proprietary 
butavailable  to 

common  viewer 

Accessible  through 

open  source 

applications 

j  Understandability 

Simplicity 

Connections 

7 

What  percentage  of  products  contain  links 
between  entities  that  are  easy  to  understand? 

Percentage 

Architecture  Redundancy 

8 

What  is  the  ratio  of  unnecessarydupli cation  per 
items  of  information? 

>  1  :  10 

Be  twee  n  1  :  10  a  nd  1  : 

100 

Betwee n  1:  100  a  nd  1: 

500 

Between  0  a  nd  1: 

500 

Architecture  Economy 

9 

Are  multiple  steps  unnecessa  ri  ly  bei  ng  used  to 
represent  the  same  activity? 

No 

Yes 

Readability 

OV  Readability 

10 

What  percentage  of  Operationa  1  Views  a  re 
presented  clearlyand  concisely? 

Percentage 

SV  Readability 

11 

What  percentage  of  System  Views  are  presented 
clearlyand  concisely? 

Percentage 

1  1 

Modifiability 


Scalability 

Scale 

12 

Can  architecture  scale  be  doubled 
while  retaining  its  desired  function 

No  views  coul d  be 

doubled 

Some  views  could 

be  doubled 

Most  views  could 

be  doubled 

All  views  could  be 

doubled 

Tailorability 

Decomposition 

13 

How  many  levels  of  de  comp  os  iti  on  are 
present  i n  OV-5? 

None 

1  level 

2  1  e  ve  1  s 

3+  levels 

Evolvability 

Tool  Format 

14 

In  genera  1,  to  what  degree  a  re 
products  developed  with  a  tool  that 
enforces  DoDAF  view  consistency  a  nd 
allows  for  easy  editi ng? 

The  product  has  to 
be  completely 
re  b  u  i  1 1 

One  input  gets 
reflected  in  single 
reference  (e.g.  no 
find  and  replace  in 
ppt) 

One  input  gets 
reflected  in  instant 

view  references 

but  not  other  views 
(e.g.  word) 

One  input  gets 
reflected  in  all 
rel eva nt  views  (e.g. 
System  Architect) 

4^ 

oo 


4^ 


Accountability 


Compliancy 

DoDAF  Compliancy 

15 

What  percentage  of  architecture 
products  complywith  DoDAF 
sta  nda  rds? 

Percentage 

Traceability 

Traceability 

16 

What  percentage  of  requirements  are 
met  by  functions/activities  (evaluate 
SV-5)? 

Percentage 

Consistency 

Internal  Consistency 

17 

What  percentage  of  available 
architecture  products  have  no  internal 
inconsistencies? 

Percentage 

External  Consistency 

18 

What  percentage  of  available 
architecture  products  have  no  external 
inconsistencies? 

Percentage 

SME  Input 

SME  Effectiveness 

19 

How  effective  are  SME's  in  architecture 
development? 

No  Plan 

SME  Involvement 

20 

How  manySMEs  across  different 
sta keholder  orga nizations  are 
involved  with  architecture 

None 

PI  a  n  exists,  but  no 
SME's  identified 


SME's  identified 


Id'd  SME's  with 
average  <5  yrs  exp 


Id'd  SME's  with 
average  >5  yrs  exp 


One  stakeholder 

orga  nization 

Two  sta kehol der 

orga  nizations 

Three  stakeholder 

orga  nizations 

More  tha n  four 

sta  kehol  der 

orga  nizations 

Appendix  C.  Measure  Summary  Table 

This  table  provides  a  summary  of  the  Architecture  Quality  Value  branch  measures. 


Table  14.  Measure  Summary  Table 


Value 

Measure 

SDVF  Type 

Min 

Max 

Subs  crib  ability 

ACCESS 

Category 

No  Access 

Access  <  5  min. 

Subs  crib  ability 

PRODUCT 

LOCATABILITY 

Category 

Cannot  locate 

<  5  min  to  locate 

Protectability 

ACCESS  CONTROL 

Category 

No  protection/  No  plan 

Appropriate  protection 

Controllability 

DOCUMENT 

PROTECTION 

Category 

No  write  protection 

All  products  controlled 

Longevity 

FILE 

MANAGEMENT 

Category 

No  system 

Current  system 

Longevity 

FILE  FORMAT 

Category 

Not  electronic 

General  File  Format 

Simplicity 

CONNECTIONS 

(Percentage) 

Monotonically 

Increasing 

Exponential 

0% 

100% 

Simplicity 

ARCHITECTURE 

REDUNDANCY 

Category 

1  found  in  <10  entities 

1  found  in  >500  entities 

Simplicity 

ARCHITECTURE 

ECONOMY 

Binary 

Yes  (found) 

None  found 

Readability 

OV  READABILITY 
(Percentage) 

Monotonically 

Increasing 

S-Curve 

Not  Readable 

All  Easy  to  Read 

Readability 

SV  READABILITY 
(Percentage) 

Monotonically 

Increasing 

S-Curve 

Not  Readable 

All  Easy  to  Read 

Scalability 

SCALE 

Category 

No  views  can  be  scaled 
2X 

All  views  can  be  scaled 
2X 

Tailorability 

DECOMPOSITION 

Category 

None 

3+  levels 

Evolvability 

TOOL  FORMAT 

Category 

Complete  product 
rebuild 

One  input  carries  thru 
multi  views 

Compliancy 

DODAF 

COMPLIANCY 

(Percentage) 

Monotonically 

Increasing 

Linear 

0% 

100% 

Traceability 

REQUIREMENT 

TRACEABILITY 

(Percentage) 

Monotonically 

Increasing 

Exponential 

0% 

(No  SV-5) 

100% 

(Complete,  Validated 
SV-5) 

Consistency 

INTERNAL 

CONSISTENCY 

(Percentage) 

Monotonically 

Increasing 

S-Curve 

0% 

100% 

Consistency 

EXTERNAL 

CONSISTENCY 

(Percentage) 

Monotonically 

Increasing 

S-Curve 

0% 

100% 

SME  Input 

SME 

EFFECTIVENESS 

Category 

No  plan  to  involve 
SME’s 

SMEs  id’d  with  5+  yrs. 
experience 

SME  Input 

SME 

INVOLVEMENT 

Category 

None 

Multiple  SME’s/ 
multiple  orgs 
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Appendix  D.  System  Effectiveness  Weighted  Hierarchy 


Figure  59  shows  the  Tier  I  System  Effectiveness  Value  branch  hierarchy  with  local  and 
global  weights. 


Figure  59.  Weighted  System  Effectiveness  Value  Branch 
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Appendix  E.  Measurement  Analysis  Graphs 


As  discussed  in  Chapter  IV,  the  following  graphs  represent  the  measurement  analyses 
performed  by  varying  the  results  for  each  measure  from  lowest  possible  assessment  to  highest 
possible  assessment.  The  graphs  are  presented  in  order. 


JFPASS  Access  Change  4 

0.746 

JFPASS  Access  Change  3 

0.732 

JFPASS  (As  Evaluated) 

0.718 

JFPASS  Access  Change  2 

0.704 

JFPASS  Access  Change  1 

0.690 

□  Access  Control 

□  OV  Readability 

□  Decomposition 

■  Document  Protection 
□  SV  Readability 

■  Access 


m 

i 


□  DoDAF  Compliancy 
■  Scale 

□  File  Management 


□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


□  Requirement  Traceability  DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  60.  Access  Measurement  Analysis 
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JFPASS  (As  Evaluated)  0.718 


JFPASS  Product  Locatability  Change 2  0.704 


JFPASS  Product  Locatability  Change  1  0.690 


□  Access  Control 

□  OV  Readability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


□  Document  Protection  □  DoDAF  Compliancy 

□  SV  Readability  B Scale 

■  Access  □  File  Management 

□  Requirement  Traceability  E3 SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  61.  Product  Locatability  Measurement  Analysis 


JFPASS  (As  Evaluated)  0.718 


JFPASS  Access  Control  Change  2  0.655 


JFPASS  Access  Control  Change  1  0.634 


□  Access  Control 

□  OV  Readability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


□  Document  Protection  □  DoDAF  Compliancy 

□  SV  Readability  B Scale 

■  Access  □  FileManagement 

□  Requirement  Traceability  0! SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  62.  Access  Control  Measurement  Analysis 
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JFPASS  (As  Evaluated)  0.718 


JFPASS  Document  Protection  Change  2  0.655 


JFPASS  Document  Protection  Change  1  0.634 


■  Access  Control 

■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  E  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  File  Management 

□  Requirement  Traceability □  SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  63.  Document  Protection  Measurement  Analysis 


JFPASS  File  Management  Change2  0.770 


JFPASS  File  Management  Change  1 


0.744 


I 


JFPASS  (As  Evaluated)  0.718 


□  Access  Control 

■  OVReadability 

□  Decomposition 

■  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  B  DoDAF  Compliancy 

□  SV  Readability  DScale 

■  Access  □  FileManagement 

□  Requirement  Traceability DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  64.  File  Management  Measurement  Analysis 
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JFPASS  (As  Evaluated)  0.718 


JFPASS  File  Format  Change  2  0.678 


JFPASS  FileFormat  Change  1  0.665 


■  Access  Control 

■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  E  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  File  Management 

□  Requirement  Traceability □  SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  65.  File  Format  Measurement  Analysis 


JFPASS  Connections  Change  12 

0.730 

JFPASS  Connections  Change  11 

0.723 

JFPASS  (As  Evaluated) 

0.718 

JFPASS  Connections  Change  10 

0.713 

JFPASS  Connections  Change  9 

0.710 

JFPASS  Connections  Change  8 

0.707 

JFPASS  Connections  Change  7 

0.705 

JFPASS  Connections  Change  6 

0.703 

JFPASS  Connections  Change  5 

0.701 

JFPASS  Connections  Change  4 

0.700 

JFPASS  Connections  Change  3 

0.699 

JFPASS  Connections  Change  2 

0.698 

JFPASS  Connections  Change  1 

0.698 

i  i  i  i  i  i 


. . 

. 

. 


. 

i  i  i  i  i  i 


~i  i  i 


ZD 


TJD 


TJD 


□  Access  Control 

■  OVReadability 

□  Decomposition 

■  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  B  DoDAF  Compliancy 

□  SV  Readability  DScale 

■  Access  □  FileManagement 

□  Requirement  Traceability DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  66.  Connections  Measurement  Analysis 
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JFPASS  (As  Evaluated)  0.718 


n 


JFPASS  Architecture  Redundancy  Change  3  0.701 


JFPASS  Architecture  Redundancy  Change  2  0.692 


JFPASS  Architecture  Redundancy  Change  1  0.685 


1 


1 


■  Access  Control 

■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  E  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  File  Management 

□  Requirement  Traceability □  SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  67.  Architecture  Redundancy  Measurement  Analysis 


JFPASS  (As  Evaluated) 


JFPASS  Architecture  Economy  Change  1 


□  Access  Control 

■  OVReadability 

□  Decomposition 

■  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  B  DoDAF  Compliancy 

□  SV  Readability  DScale 

■  Access  □  FileManagement 

□  Requirement  Traceability DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  68.  Architecture  Economy  Measurement  Analysis 
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JFPASS  OV  Readability  Change 5 

JFPASS  (As  Evaluated) 

JFPASS  OV  Readability  Change 4 

JFPASS  OV  Readability  Change 3 

JFPASS  OV  Readability  Change 2 

JFPASS  OV  Readability  Change  1 


■  Access  Control 

■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  E  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  File  Management 

□  Requirement  Traceability DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  69.  OV  Readability  Measurement  Analysis 


JFPASS  SV  Readability Change5 

JFPASS  SV  Readability  Change  4 

JFPASS  (As  Evaluated) 

JFPASS  SV  Rea  da  bi  I  i  ty  Cha  nge  3 

JFPASS  SV  Readability  Cha  nge  2 

JFPASS  SV  Rea  da  bi  I  i  ty  Cha  nge  1 


■  Access  Control 

□  OVReadability 

□  Decomposition 

■  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


□  Document  Protection  B  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  File  Management 

□  Requirement  Traceability  II SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  70.  SV  Readability  Measurement  Analysis 
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JFPASS  ScaleChange  3  0.742 


JFPASS  (As  Evaluated)  0.718 


JFPASS  ScaleChange  2  0.700 


JFPASS  ScaleChange  1  0.682 


■  Access  Control 

■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  E  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  File  Management 

□  Requirement  Traceability □  SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  71.  Scale  Measurement  Analsysis 


JFPASS  (As  Evaluated) 


JFPASS  Decomposition  Change  3 


JFPASS  Decomposition  Change  2 


JFPASS  Decomposition  Change  1 


□  Access  Control 
■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  □  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  FileManagement 

□  Requirement  Traceability DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  72.  Decomposition  Measurement  Analysis 
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JFPASS  (As  Evaluated)  0.718 


JFPASS  Tool  Format  Change  3  0.706 


JFPASS  Tool  Format  Change  2  0.700 


JFPASS  Tool  Format  Change  1  0.688 


■  Access  Control 

■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  E  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  File  Management 

□  Requirement  Traceability □  SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  73.  Tool  Format  Measurement  Analysis 


JFPASS  DoDAF  Compliancy Changel2  0.730 
JFPASS  DoDAF  Compliancy Changell  0.724 
JFPASS  (As  Evaluated)  0.718 
JFPASS  DoDAF  Compliancy ChangelO  0.712 
JFPASS  DoDAF  Compliancy Change9  0.705 
JFPASS  DoDAF  Compliancy Change8  0.699 
JFPASS  DoDAF  Compliancy  Change  7  0.693 
JFPASS  DoDAF  Compliancy Change6  0.687 
JFPASS  DoDAF  Compliancy Change5  0.680 
JFPASS  DoDAF  Compliancy Change4  0.674 
JFPASS  DoDAF  Compliancy Change3  0.668 
JFPASS  DoDAF  Compliancy Change2  0.662 
JFPASS  DoDAF  Compliancy  Change  1  0.655 
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□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  □  DoDAF  Compliancy 
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□  External  Consistency 


Figure  74.  DoDAF  Compliancy  Measurement  Analysis 
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JFPASS  Requirement  Traceability ChangelO  0.768 
JFPASS  Requirement  Traceability Change9  0.752 
JFPASS  Requirement  Traceability Change8  0.741 
JFPASS  Requirement  Traceability  Change  7  0.733 
JFPASS  Requirement  Traceability Change6  0.728 
JFPASS  Requirement  Traceability Change5  0.724 
JFPASS  Requirement  Traceability Change4  0.722 
JFPASS  Requirement  Traceability Change3  0.720 
JFPASS  Requirement  Traceability Change2  0.719 
JFPASS  Requirement  Traceability  Change  1  0.718 
JFPASS  (As  Evaluated)  0.718 


I  I  I  I  I  I  I  I 
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■  OVReadability 
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Figure  75.  Requirement  Traceability  Measurement  Analysis 


JFPASS  Internal  Consistency  Change  12  0.719 
JFPASS  Internal  Consistency  Change  11  0.719 
JFPASS  (As  Evaluated)  0.718 
JFPASS  Internal  Consistency  Change  10  0.717 
JFPASS  Internal  Consistency  Change  9  0.715 
JFPASS  Internal  Consistency  Change  8  0.712 
JFPASS  Internal  Consistency  Change  7  0.707 
JFPASS  Internal  Consistency  Change  6  0.702 
JFPASS  Internal  Consistency  Change  5  0.699 
JFPASS  Internal  Consistency  Change  4  0.697 
JFPASS  Internal  Consistency  Change  3  0.695 
JFPASS  Internal  Consistency  Change  2  0.695 
JFPASS  Internal  Consistency  Change  1  0.694 


□  Access  Control 

■  OVReadability 

□  Decomposition 

■  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 
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□  SV  Readability  DScale 

■  Access  □  FileManagement 

□  Requirement  Traceability DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  76.  Internal  Consistency  Measurement  Analysis 
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JFPASS  External  Consistency  Change  12  0.719 
JFPASS  External  Consistency  Change  11  0.719 
JFPASS  (As  Evaluated)  0.718 
JFPASS  External  Consistency  Change  10  0.717 
JFPASS  External  Consistency  Change  9  0.715 
JFPASS  External  Consistency  Change  8  0.712 
JFPASS  External  Consistency  Change  7  0.707 
JFPASS  External  Consistency  Change  6  0.702 
JFPASS  External  Consistency  Change  5  0.699 
JFPASS  External  Consistency  Change  4  0.697 
JFPASS  External  Consistency  Change  3  0.695 
JFPASS  External  Consistency  Change  2  0.695 
JFPASS  External  Consistency  Change  1  0.694 


■  Access  Control 

■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  E  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  File  Management 

□  Requirement  Traceability □  SME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  77.  External  Consistency  Measurement  Analysis 


JFPASS  SME  Effectiveness  Change  4 


JFPASS  SME  Effectiveness  Change  3 


JFPASS  SME  Effectiveness  Change  2 


JFPASS  SME  Effectiveness  Change  1 


JFPASS  (As  Evaluated) 


□  Access  Control 

■  OVReadability 

□  Decomposition 

■  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  B  DoDAF  Compliancy 

□  SV  Readability  DScale 

■  Access  □  FileManagement 

□  Requirement  Traceability DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  78.  SME  Effectiveness  Measurement  Analysis 
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JFPASS  SME  Involvement  Change  5  0.755 

JFPASS  SME  Involvement  Change  4  0.748 

JFPASS  SME  Involvement  Change  3  0.731 

JFPASS  SME  Involvement  Change  2  0.723 

JFPASS  SME  Involvement  Change  1  0.722 

JFPASS  (As  Evaluated)  0.718 


■  Access  Control 

■  OVReadability 

□  Decomposition 

□  FileFormat 

□  SME  Involvement 

□  Architecture  Economy 

□  Internal  Consistency 


■  Document  Protection  E  DoDAF  Compliancy 

□  SV  Readability  BScale 

■  Access  □  FileManagement 

□  Requirement  Traceability DSME  Effectiveness 

□  Connections  □  Architecture  Redundancy 

□  Tool  Format  □  ProductLocatability 

□  External  Consistency 


Figure  79.  SME  Involvement  Measurement  Analysis 
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Appendix  F.  Weight  Sensitivity  Analysis  Summary  Tables 


Tables  15  through  18  summarize  the  weight  sensitivity  analysis  results  for  each  of  the 
values  and  measures.  The  ‘Max  Negative  Change’  and  ‘Max  Positive  Change’  columns 
represent  how  the  overall  score  would  be  affected  if  the  weight  was  adjusted  in  either  direction. 
For  example,  the  value  Accessibility  has  a  local  weight  of  0.250.  Decreasing  its  weight  towards 
zero  would  eliminate  it  as  one  of  the  second-tier  values  and  lower  the  overall  possible 
Architecture  Quality  Values  from  0.718  to  0.660  points.  Increasing  the  weight  to  one,  thus 
making  Accessibility  the  only  second-tier  value,  would  raise  the  overall  Architecture  Quality 
Values  score  to  0.880  points.  Note  that  no  change  occurred  for  REQUIREMENT  TRACEABILITY, 
SME  EFFECTIVENESS,  and  SME  INVOLVEMENT  because  their  parent  values  of  Traceability  and 
SME  Input  both  earned  zero  value.  Similarly,  the  results  for  Consistency  did  not  change  because 
INTERNAL  and  EXTERNAL  CONSISTENCY  scored  the  same  and  had  equal  weights. 


Table  15.  Accessibility  Sensitivity  Results 


Assigned 

^ i-local 

Sensitivity 

Line 

Slope 

Max  Negative  Effect 

Max  Positive  Effect 

^i— local 

II  VAQ-New 
VAQ-Base  II 

^i— local 

II  VAQ-New 
VAQ-Base  II 

Accessibility 

0.250 

+0.220 

0.000 

-0.060 

1.000 

+0.160 

Subscribability 

0.330 

-0.090 

1.000 

-0.070 

0.000 

+0.020 

Access 

0.667 

-0.040 

1.000 

-0.020 

0.000 

+0.020 

Product 

Locatability 

0.333 

+0.050 

0.000 

-0.020 

1.000 

+0.030 

Controllability 

0.330 

+0.040 

0.000 

-0.020 

1.000 

+0.020 

Document 

Protection 

1.000 

+0.080 

0.000 

-0.080 

1.000 

No  Change 

Protectability 

0.330 

+0.040 

0.000 

-0.020 

1.000 

+0.020 

Access 

Control 

1.000 

+0.080 

0.000 

-0.080 

1.000 

No  Change 
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Table  16.  Usability  Sensitivity  Results 


Assigned 

^-i-local 

Sensitivity 

Line 

Slope 

Max  Negative  Effect 

Max  Positive  Effect 

^ i-local 

II  VAQ-New 
VAQ-Base  II 

^i-local 

II  VAQ-New 
VAQ-Base  II 

Usability 

0.350 

+0.038 

0.000 

-0.014 

1.000 

+0.024 

Longevity 

0.300 

-0.130 

1.000 

-0.100 

0.000 

+0.030 

File 

Management 

0.500 

-0.100 

1.000 

-0.060 

0.000 

+0.040 

File  Format 

0.500 

+0.100 

0.000 

-0.060 

1.000 

+0.040 

Understandabilty 

0.700 

+0.120 

0.000 

-0.080 

1.000 

+0.040 

Simplicity 

0.400 

+0.020 

0.000 

-0.010 

1.000 

+0.010 

Connections 

0.330 

-0.040 

1.000 

-0.030 

0.000 

+0.010 

Architecture 

Redundancy 

0.330 

+0.020 

0.000 

-0.010 

1.000 

+0.010 

Architecture 

Economy 

0.330 

+0.020 

0.000 

-0.010 

1.000 

+0.010 

Readability 

0.600 

-0.020 

1.000 

-0.010 

0.000 

+0.010 

OV 

Readability 

0.500 

+0.040 

0.000 

-0.020 

1.000 

+0.020 

SV 

Readability 

0.500 

-0.030 

1.000 

-0.020 

0.000 

+0.010 

Table  17.  Modifiability  Sensitivity  Results 


Assigned 

^ i-local 

Sensitivity 
Line  Slope 

Max  Negative  Effect 

Max  Positive  Effect 

^ i-local 

II  VAQ-New 

^AQ-Base  II 

^ i-local 

II 

VAQ-Base  II 

Modifiability 

0.150 

+0.140 

0.000 

-0.020 

1.000 

+0.120 

Scalability 

0.400 

-0.060 

1.000 

-0.020 

0.000 

+0.030 

Scale 

1.000 

+0.040 

0.000 

-0.040 

1.000 

No  Change 

Tailorability 

0.400 

+0.040 

0.000 

-0.020 

1.000 

+0.020 

Decomposition 

1.000 

+0.060 

0.000 

-0.060 

1.000 

No  Change 

Evolvability 

0.200 

+0.030 

0.000 

-0.010 

1.000 

+0.020 

Tool  Format 

1.000 

+0.030 

0.000 

-0.030 

1.000 

No  Change 
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Table  18.  Accountability  Sensitivity  Results 


Assigned 

^ i-local 

Sensitivity 

Line 

Slope 

Max  Negative  Effect 

Max  Positive  Effect 

^i— local 

II  VAQ-New 

V AQ-Base  II 

^i-local 

II  VAQ-New 

L AQ-Base  II 

Accountability 

0.250 

-0.370 

1.000 

-0.270 

0.000 

+0.080 

Compliancy 

0.300 

+0.140 

0.000 

-0.040 

1.000 

+0.100 

DoDAF 

Compliancy 

1.000 

+0.070 

0.000 

-0.070 

1.000 

No  Change 

Traceability 

0.200 

-0.130 

1.000 

-0.100 

0.000 

+0.030 

Requirements 

Traceability 

1.000 

0.000 

0.000 

No  Change 

1.000 

No  Change 

Consistency 

0.200 

+0.140 

0.000 

-0.030 

1.000 

+0.110 

Internal 

Consistency 

0.500 

0.000 

0.000 

No  Change 

1.000 

No  Change 

External 

Consistency 

0.500 

0.000 

0.000 

No  Change 

1.000 

No  Change 

SME  Input 

0.300 

-0.160 

1.000 

-0.100 

0.000 

+0.060 

SME  Effectiveness 

0.500 

0.000 

0.000 

No  Change 

1.000 

No  Change 

SME  Involvement 

0.500 

0.000 

0.000 

No  Change 

1.000 

No  Change 
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